-- Logs begin at Sun 2022-09-11 00:10:31 UTC, end at Sun 2022-09-11 01:11:40 UTC. -- Sep 11 00:37:56.903759 dv-cinder-master-3-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Sep 11 00:37:56.912208 dv-cinder-master-3-iscsi devstack@keystone.service[87911]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Sep 11 00:37:56.914184 dv-cinder-master-3-iscsi devstack@keystone.service[87911]: *** Starting uWSGI 2.0.18-debian (64bit) on [Sun Sep 11 00:37:56 2022] *** Sep 11 00:37:56.914184 dv-cinder-master-3-iscsi devstack@keystone.service[87911]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Sep 11 00:37:56.914184 dv-cinder-master-3-iscsi devstack@keystone.service[87911]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Sep 11 00:37:56.914184 dv-cinder-master-3-iscsi devstack@keystone.service[87911]: nodename: dv-cinder-master-3-iscsi Sep 11 00:37:56.914184 dv-cinder-master-3-iscsi devstack@keystone.service[87911]: machine: x86_64 Sep 11 00:37:56.914184 dv-cinder-master-3-iscsi devstack@keystone.service[87911]: clock source: unix Sep 11 00:37:56.914184 dv-cinder-master-3-iscsi devstack@keystone.service[87911]: pcre jit disabled Sep 11 00:37:56.914184 dv-cinder-master-3-iscsi devstack@keystone.service[87911]: detected number of CPU cores: 4 Sep 11 00:37:56.914184 dv-cinder-master-3-iscsi devstack@keystone.service[87911]: current working directory: / Sep 11 00:37:56.914184 dv-cinder-master-3-iscsi devstack@keystone.service[87911]: detected binary path: /usr/bin/uwsgi-core Sep 11 00:37:56.914184 dv-cinder-master-3-iscsi devstack@keystone.service[87911]: your processes number limit is 47680 Sep 11 00:37:56.914184 dv-cinder-master-3-iscsi devstack@keystone.service[87911]: your memory page size is 4096 bytes Sep 11 00:37:56.914851 dv-cinder-master-3-iscsi devstack@keystone.service[87911]: detected max file descriptor number: 2048 Sep 11 00:37:56.914851 dv-cinder-master-3-iscsi devstack@keystone.service[87911]: lock engine: pthread robust mutexes Sep 11 00:37:56.914851 dv-cinder-master-3-iscsi devstack@keystone.service[87911]: thunder lock: enabled Sep 11 00:37:56.914851 dv-cinder-master-3-iscsi devstack@keystone.service[87911]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Sep 11 00:37:56.914851 dv-cinder-master-3-iscsi devstack@keystone.service[87911]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Sep 11 00:37:56.954039 dv-cinder-master-3-iscsi devstack@keystone.service[87911]: Python main interpreter initialized at 0x55a40aea79c0 Sep 11 00:37:56.954039 dv-cinder-master-3-iscsi devstack@keystone.service[87911]: python threads support enabled Sep 11 00:37:56.954039 dv-cinder-master-3-iscsi devstack@keystone.service[87911]: your server socket listen backlog is limited to 100 connections Sep 11 00:37:56.954039 dv-cinder-master-3-iscsi devstack@keystone.service[87911]: your mercy for graceful operations on workers is 90 seconds Sep 11 00:37:56.954423 dv-cinder-master-3-iscsi devstack@keystone.service[87911]: mapped 403077 bytes (393 KB) for 2 cores Sep 11 00:37:56.954506 dv-cinder-master-3-iscsi devstack@keystone.service[87911]: *** Operational MODE: preforking *** Sep 11 00:37:56.954733 dv-cinder-master-3-iscsi devstack@keystone.service[87911]: *** uWSGI is running in multiple interpreter mode *** Sep 11 00:37:56.954733 dv-cinder-master-3-iscsi devstack@keystone.service[87911]: spawned uWSGI master process (pid: 87911) Sep 11 00:37:56.954951 dv-cinder-master-3-iscsi systemd[1]: Started Devstack devstack@keystone.service. Sep 11 00:37:56.956042 dv-cinder-master-3-iscsi devstack@keystone.service[87911]: spawned uWSGI worker 1 (pid: 87923, cores: 1) Sep 11 00:37:56.956042 dv-cinder-master-3-iscsi devstack@keystone.service[87911]: spawned uWSGI worker 2 (pid: 87924, cores: 1) Sep 11 00:37:56.956042 dv-cinder-master-3-iscsi devstack@keystone.service[87911]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Sep 11 00:38:01.233546 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 11 00:38:01.233546 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: from cryptography.utils import int_from_bytes Sep 11 00:38:01.234324 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 11 00:38:01.234324 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: from cryptography.utils import int_from_bytes Sep 11 00:38:01.234767 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 11 00:38:01.234767 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: from cryptography.utils import int_from_bytes Sep 11 00:38:01.234938 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 11 00:38:01.234938 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: from cryptography.utils import int_from_bytes Sep 11 00:38:01.284440 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Sep 11 00:38:01.284895 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Sep 11 00:38:01.284988 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Sep 11 00:38:01.285078 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Sep 11 00:38:01.285158 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] command line args: [] {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Sep 11 00:38:01.285230 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] command line args: [] {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Sep 11 00:38:01.285309 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Sep 11 00:38:01.285371 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Sep 11 00:38:01.285496 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] ================================================================================ {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Sep 11 00:38:01.285627 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] ================================================================================ {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Sep 11 00:38:01.285744 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] admin_token = **** {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.285837 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] admin_token = **** {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.285985 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] config_dir = [] {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.286072 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] config_dir = [] {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.286152 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.286223 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.286333 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] config_source = [] {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.286424 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] config_source = [] {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.286498 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] debug = True {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.286568 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] debug = True {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.286688 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.287231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.287231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.287395 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.287456 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] insecure_debug = False {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.287518 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] insecure_debug = False {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.287577 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.287642 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.287707 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.287771 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.287951 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] list_limit = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.288051 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] list_limit = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.288051 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] log_config_append = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.288246 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] log_config_append = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.288351 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.288454 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.288551 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] log_dir = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.288654 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] log_dir = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.288756 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] log_file = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.288864 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] log_file = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.288970 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.289049 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.289167 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.289250 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.289331 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.289460 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.289543 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.289617 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.289735 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.289813 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.289945 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.290024 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.290142 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.290225 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.290350 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.290437 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.290518 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.290640 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.290717 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.290810 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.290926 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.291040 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.291186 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.291264 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.291390 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.291468 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.291631 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] notification_format = cadf {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.291766 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] notification_format = cadf {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.291986 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.292171 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.292438 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.27/identity {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.292522 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.27/identity {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.292606 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] publish_errors = False {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.292730 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] publish_errors = False {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.292837 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.292915 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.293063 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.293141 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.293239 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.293324 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.293480 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.293615 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.293702 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.293826 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.293928 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] standard_threads = False {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.294006 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] standard_threads = False {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.294156 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] strict_password_check = False {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.294235 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] strict_password_check = False {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.294394 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.294471 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.294566 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] use_eventlog = False {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.294643 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] use_eventlog = False {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.294774 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] use_journal = False {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.294873 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] use_journal = False {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.294946 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] use_json = False {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.295027 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] use_json = False {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.295100 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] use_stderr = False {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.295171 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] use_stderr = False {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.295297 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] use_syslog = False {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.295381 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] use_syslog = False {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.295460 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] watch_log_file = False {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.295532 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] watch_log_file = False {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 11 00:38:01.295692 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.295771 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.295928 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.296007 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.296180 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.296265 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.296418 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.296501 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.296661 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.296739 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.296894 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.296972 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.297139 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.297217 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.297369 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.297447 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.297612 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.297690 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.297853 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.297930 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.298097 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.298183 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.298328 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.298404 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.298566 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.298644 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.298828 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.298921 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.299073 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.299158 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.299314 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.299401 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.299546 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.299626 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.299796 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.299873 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.300046 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.300124 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.300273 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.300348 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.300509 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.300587 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.300760 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.300838 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.301016 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.301095 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.301266 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.301354 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.301509 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.301598 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.301722 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.301801 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.301953 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.302032 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.302181 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.302266 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.302410 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.302487 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.302654 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.302731 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.302923 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.303015 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.303151 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.303235 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.303378 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.303470 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.303618 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.303708 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.303862 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.303947 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.304092 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.304169 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.304333 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.304421 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.304560 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.304639 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.304789 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.304874 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.305022 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.305107 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.305245 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.305322 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.305471 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.305558 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.305697 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.305849 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.305926 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.305995 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.306140 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.306225 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.306383 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.306470 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.306610 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.306689 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.306863 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.306946 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.307092 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.307746 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.307887 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.307975 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.308125 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.308358 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.308464 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.308579 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.308677 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.308868 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.308981 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.309060 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.309122 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.309268 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.309354 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.309497 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.309671 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.309758 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.309842 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.309946 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.310074 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.310158 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.310230 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.310403 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.310494 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.310636 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.310756 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.310890 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.310985 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.311119 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.311207 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.311375 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.311466 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.311607 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.311686 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.311837 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.311920 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.312071 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.312160 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.312301 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.312380 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.312561 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.312642 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.312794 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.312883 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.313037 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.313158 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.313273 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.313354 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.313505 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.313592 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.313737 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.313815 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.313966 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.314045 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] cache.enabled = True {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.314204 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.314293 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] cache.enabled = True {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.314434 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.314512 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.314666 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.314745 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.314923 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.315013 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.315153 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.315231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.315391 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.315469 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.315622 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.315709 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.315852 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.315930 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.316080 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.316165 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.316309 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.316397 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.316539 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.316619 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.316792 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.316870 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.317024 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.317109 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.317264 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.317349 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.317499 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.317577 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.317737 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.317893 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.317971 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.318043 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.318194 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.318273 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.318425 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.318512 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.318653 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.318733 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.318902 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.318993 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.319131 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.319211 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.319359 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.319446 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.319587 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.319665 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.319813 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.319891 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.320045 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.320125 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.320276 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.320354 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.320513 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.320592 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.320741 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.320820 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.320992 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.321071 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.321237 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.321315 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.321470 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.321560 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] auth.external = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.321699 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] auth.external = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.321791 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] auth.mapped = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.321926 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] auth.mapped = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.322007 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.322206 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.322292 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.322432 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.322512 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] auth.password = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.322660 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] auth.token = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.322741 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] auth.password = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.323490 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.323585 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] auth.token = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.323730 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] catalog.caching = True {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.323960 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.324040 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.324198 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.324281 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] catalog.caching = True {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.324444 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.324606 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.324687 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.324760 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.324903 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.324991 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.325138 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] credential.caching = True {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.325226 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.325363 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.325442 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] credential.driver = sql {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.325597 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] credential.caching = True {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.325685 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.325840 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] credential.driver = sql {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.325929 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.326079 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.326166 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.326318 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.326399 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.326550 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.326639 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.326837 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.326923 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.327038 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.327120 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.327280 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.327368 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.327526 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.327605 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.327757 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.327838 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.328003 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.328091 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.328279 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.328359 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.328516 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.328596 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.328771 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.328860 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.329003 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.329090 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] federation.caching = True {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.329236 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.329315 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.329472 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] federation.caching = True {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.329559 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] federation.driver = sql {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.329704 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.329785 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.329933 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.330013 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] federation.driver = sql {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.330176 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.330263 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.330401 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.330483 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.330645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.330726 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.330901 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.330983 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.331150 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.331230 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.331389 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.331475 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.331625 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.331704 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.331855 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] identity.caching = True {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.331932 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.332093 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.332172 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.332326 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] identity.caching = True {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.332403 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.332554 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.332635 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.332784 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.332863 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.333012 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.333092 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] identity.driver = sql {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.333242 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.333321 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.333476 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.333556 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] identity.driver = sql {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.333706 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.333785 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.333936 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.334015 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.334170 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.334250 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.334398 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.334479 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.334623 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.334702 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.334872 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.334957 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.335100 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.335182 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.335338 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.335418 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.335578 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.335658 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.335820 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.335901 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.336066 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.336145 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.336306 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.336388 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.336537 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.336616 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.336765 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.336850 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.336997 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.337077 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.337220 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.337300 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.337450 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.337528 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.337672 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.337751 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.337900 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.337983 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.338135 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.338215 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.338360 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.338464 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.338632 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.338773 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.338907 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.339103 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.339185 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.339353 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.339447 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.339602 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.339704 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.339825 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.339904 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.340057 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.340148 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.340269 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] ldap.password = **** {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.340460 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.340542 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.340694 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.340782 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.340928 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.341009 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.341155 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] ldap.password = **** {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.341235 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.341390 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.341507 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.341624 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.341758 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.341860 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.342005 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.342092 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.342238 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.342426 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.342507 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.342679 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.342759 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.342953 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.343046 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.343188 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.343268 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.343417 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.343496 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.343651 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.343739 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.343882 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] ldap.user = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.343963 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.344122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.344203 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.344355 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.344443 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.344589 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.344671 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.344818 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] ldap.user = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.344898 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.345055 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.345146 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.345296 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.345375 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.345528 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.345607 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.345755 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.345842 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.345989 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.346070 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.346223 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.346303 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.346453 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.346540 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.346680 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.346759 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.346935 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.347017 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.347169 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.347262 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.347400 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.347481 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.347627 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.347707 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.347869 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.347957 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.348096 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.348176 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.348340 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.348421 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.348574 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.348662 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.348806 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.348888 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.349030 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.349109 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.349259 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.349347 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.349491 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.349571 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.349722 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.349802 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.349953 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.350042 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.350182 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.350312 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] policy.driver = sql {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.350417 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.350497 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.350658 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.350747 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.350912 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.350994 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.351144 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] receipt.caching = True {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.351223 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] policy.driver = sql {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.351377 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.351468 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.351615 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.351694 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.351845 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.351925 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.352074 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.352162 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] receipt.caching = True {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.352344 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.352431 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.352548 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] resource.caching = True {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.352625 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.352750 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.352837 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.352982 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.353063 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] resource.driver = sql {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.353212 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.353293 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.353448 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] resource.caching = True {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.353537 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.353692 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.353772 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.353926 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] revoke.caching = True {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.354006 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] resource.driver = sql {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.354194 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.354276 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.354429 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.354677 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.355008 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] role.cache_time = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.355095 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.355245 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] revoke.caching = True {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.355400 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] role.caching = True {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.355478 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.355552 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] role.driver = sql {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.355711 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.355792 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] role.list_limit = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.355945 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] role.cache_time = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.356074 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.356178 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] role.caching = True {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.356259 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.356420 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] role.driver = sql {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.356501 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.356648 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] role.list_limit = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.356730 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.356891 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.356971 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.357133 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.357212 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.357376 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.357457 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.357617 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.357696 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.357849 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.357929 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.358088 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.358169 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.358330 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.358411 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.358573 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.358653 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.358806 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.358888 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.359063 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.359217 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.359308 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.359452 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.359542 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.359699 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.359780 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.359946 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.360139 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.360219 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.360375 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.360457 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.360621 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.360702 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.360870 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.360950 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.361116 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.361196 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.361353 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.361433 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.361580 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.361667 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.361813 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.361892 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.362042 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.362123 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.362285 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.362372 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.362514 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.362595 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.362741 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.362852 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.362993 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.363086 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.363229 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.363309 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.363457 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] token.cache_time = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.363538 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.363689 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] token.caching = True {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.363781 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.363923 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.364004 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.364152 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.364232 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] token.provider = fernet {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.364378 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] token.cache_time = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.364466 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.364610 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] token.caching = True {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.364691 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.364842 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.364922 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.365077 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.365164 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] token.provider = fernet {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.365306 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.365385 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.365545 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.365624 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.365784 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.365871 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] trust.driver = sql {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.366014 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.366095 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.366248 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.366327 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.366481 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.366574 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.366719 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] trust.driver = sql {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.366844 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.366976 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.367058 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.367210 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.367303 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.367442 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.367532 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.367684 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.367766 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.367958 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.368038 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] database.connection = **** {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.368185 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.368275 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.368422 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.368504 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.368659 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.368739 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.368899 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.368979 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] database.connection = **** {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.369133 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.369216 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.369367 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.369447 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.369601 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.370222 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.370467 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.370556 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.370708 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.370850 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.370969 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.371090 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.371205 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.371334 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.371440 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.371570 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.371683 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.371815 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.371905 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.372056 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.372137 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.372301 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.372393 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.372532 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.372623 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.372771 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.372852 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.373007 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.373088 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.373162 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=87924) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Sep 11 00:38:01.373244 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.373482 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.373715 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 11 00:38:01.373880 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=87923) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Sep 11 00:38:01.403002 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=87924) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.403228 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=87924) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.403456 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=87924) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.403544 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=87923) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.403774 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=87923) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.403999 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=87923) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.412080 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=87924) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.412496 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=87923) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.440383 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=87923) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.440610 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=87923) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.440736 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=87924) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.440892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=87924) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.493188 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=87923) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.493459 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=87924) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.542266 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=87923) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.542487 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=87923) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.542947 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=87924) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.543185 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=87924) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.700066 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=87923) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.700240 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=87924) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.700356 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=87923) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.700436 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=87923) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.700530 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=87924) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.700608 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=87923) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.700680 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=87924) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.700816 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=87923) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.700901 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=87924) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.701026 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=87923) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.701118 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=87924) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.701240 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=87924) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.723545 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=87923) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.723676 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=87924) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.723763 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=87923) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.723859 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=87924) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.723980 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=87923) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.724067 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=87924) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.724193 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=87923) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.724278 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=87924) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.724401 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=87923) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.724488 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=87924) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.724606 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=87923) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.724691 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=87924) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.724811 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=87923) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.724905 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=87924) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.736360 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=87923) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.736569 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=87924) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.748884 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=87923) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.749144 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=87923) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.749242 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=87924) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.749360 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=87923) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.749442 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=87924) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.749525 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=87923) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.749601 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=87924) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.749739 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=87923) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.749834 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=87924) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.749948 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=87923) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.750032 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=87924) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.750155 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=87923) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.750240 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=87924) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.750424 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=87924) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 11 00:38:01.767154 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.767356 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 11 00:38:01.767695 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.767889 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 11 00:38:01.768019 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.768170 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.768320 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 11 00:38:01.768456 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.768586 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.768716 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.768855 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 11 00:38:01.768987 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.769116 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.769247 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.769477 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.769736 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.779099 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.779370 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.779634 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.779909 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.780038 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.780178 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.780442 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.780723 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.787085 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=87923) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 11 00:38:01.787915 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=87924) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 11 00:38:01.789943 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=87923) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 11 00:38:01.790223 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.790502 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.790779 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.790923 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=87924) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 11 00:38:01.791079 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.791217 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.791348 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.791475 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.791640 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.791814 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.791959 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.792089 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.792237 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.792407 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.792549 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.792707 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.792925 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.793054 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.793184 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.793313 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.793442 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.815990 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.816172 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.817445 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=87924) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 11 00:38:01.817621 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=87923) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 11 00:38:01.817750 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.817888 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.822356 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=87924) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 11 00:38:01.822530 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=87923) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 11 00:38:01.822658 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.823357 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.823582 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.823794 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.829832 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=87924) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 11 00:38:01.829983 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=87923) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 11 00:38:01.830112 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.830240 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.835731 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=87924) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 11 00:38:01.835871 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=87923) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 11 00:38:01.836026 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.836176 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.836308 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.836399 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.836566 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.836649 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.836837 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.836919 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.837108 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.837192 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.837371 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.837452 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.837632 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.837714 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.853638 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.853758 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.853919 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.854008 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.857787 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=87923) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 11 00:38:01.857906 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=87924) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 11 00:38:01.861016 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.861106 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.861307 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.861392 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.865462 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=87923) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 11 00:38:01.865552 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=87924) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 11 00:38:01.868546 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=87923) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 11 00:38:01.868634 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=87924) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 11 00:38:01.871763 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.871955 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.872043 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.872254 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.872339 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.872536 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.872630 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.872816 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.872909 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.873089 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.873175 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.873364 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.888992 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=87923) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 11 00:38:01.889275 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.889457 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=87924) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 11 00:38:01.889543 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.889746 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.889845 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.890012 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.890275 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.896485 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.897051 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.897989 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.898563 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.899507 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.899782 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.900091 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.900366 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.902543 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=87923) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 11 00:38:01.902839 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.903127 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.903261 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=87924) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 11 00:38:01.903406 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.903547 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.903783 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.903868 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.904106 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.904492 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.915094 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=87923) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 11 00:38:01.915376 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.915653 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.915844 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=87924) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 11 00:38:01.915929 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.916130 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.916298 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.916412 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.917133 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.917256 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.917606 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.917692 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.917914 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.918190 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.933577 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=87923) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 11 00:38:01.934119 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=87924) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 11 00:38:01.936494 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=87923) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 11 00:38:01.937045 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=87924) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 11 00:38:01.939465 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.939944 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.940951 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.941471 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.942397 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=87923) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 11 00:38:01.942675 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.942957 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=87924) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 11 00:38:01.943087 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.943241 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.943633 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.949766 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.949991 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.951284 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=87923) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 11 00:38:01.951473 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=87924) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 11 00:38:01.954097 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.954335 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.954487 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.954735 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.954837 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.955046 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.955249 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.955339 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.962767 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=87923) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 11 00:38:01.963075 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.963173 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=87924) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 11 00:38:01.963353 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.963449 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.963675 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.970513 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=87923) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 11 00:38:01.970618 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=87924) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 11 00:38:01.970807 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.970894 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.971091 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.971175 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.971362 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.971456 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.971631 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.971715 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.971900 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.971983 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.972162 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.972255 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.972439 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.972523 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.972700 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.972785 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.972980 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.973062 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.973342 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.973436 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.973606 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.973689 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.973865 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.973948 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.974119 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=87923) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:01.974209 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=87924) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 11 00:38:02.009117 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: WSGI app 0 (mountpoint='') ready in 6 seconds on interpreter 0x55a40aea79c0 pid: 87923 (default app) Sep 11 00:38:02.012124 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: WSGI app 0 (mountpoint='') ready in 6 seconds on interpreter 0x55a40aea79c0 pid: 87924 (default app) Sep 11 00:38:02.012452 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c7eed5-d87b-4c85-bdd9-1ba5be3bb3b5 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:02.012735 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c7eed5-d87b-4c85-bdd9-1ba5be3bb3b5 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:02.013004 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c7eed5-d87b-4c85-bdd9-1ba5be3bb3b5 None None] PATH_INFO: `/v3` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:02.017117 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1/1] 10.222.0.27 () {50 vars in 791 bytes} [Sun Sep 11 00:38:02 2022] GET /identity/v3/ => generated 255 bytes in 7 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:08.908777 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e11f78-1a88-4ff5-bf87-d581bd77de6d None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:08.909299 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e11f78-1a88-4ff5-bf87-d581bd77de6d None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:08.909299 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e11f78-1a88-4ff5-bf87-d581bd77de6d None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:08.913825 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1/2] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:38:08 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:38:08.919289 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0e354d-044d-4bdc-8b6d-7300be0d214e None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:08.919754 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0e354d-044d-4bdc-8b6d-7300be0d214e None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:08.919896 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0e354d-044d-4bdc-8b6d-7300be0d214e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:08.990247 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: INFO dbcounter [None req-6f0e354d-044d-4bdc-8b6d-7300be0d214e None None] Registered counter for database keystone Sep 11 00:38:09.009340 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG oslo_db.sqlalchemy.engines [None req-6f0e354d-044d-4bdc-8b6d-7300be0d214e None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=87923) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Sep 11 00:38:09.016199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writer thread running {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Sep 11 00:38:09.194851 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG passlib.handlers.bcrypt [None req-6f0e354d-044d-4bdc-8b6d-7300be0d214e None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=87923) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Sep 11 00:38:09.195170 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG passlib.handlers.bcrypt [None req-6f0e354d-044d-4bdc-8b6d-7300be0d214e None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=87923) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Sep 11 00:38:09.253170 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-6f0e354d-044d-4bdc-8b6d-7300be0d214e None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:38:09.254814 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-6f0e354d-044d-4bdc-8b6d-7300be0d214e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:38:09.382340 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 2/3] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:38:08 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 464 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Sep 11 00:38:09.835392 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e11f78-1a88-4ff5-bf87-d581bd77de6d None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:09.905378 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: INFO dbcounter [None req-70e11f78-1a88-4ff5-bf87-d581bd77de6d None None] Registered counter for database keystone Sep 11 00:38:09.927043 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG oslo_db.sqlalchemy.engines [None req-70e11f78-1a88-4ff5-bf87-d581bd77de6d None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=87924) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Sep 11 00:38:09.933876 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writer thread running {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Sep 11 00:38:10.051208 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e11f78-1a88-4ff5-bf87-d581bd77de6d None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:10.055472 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ea8378-4987-439b-8bfc-8d0c65309c81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:10.056103 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ea8378-4987-439b-8bfc-8d0c65309c81 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:10.056380 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ea8378-4987-439b-8bfc-8d0c65309c81 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:10.056648 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ea8378-4987-439b-8bfc-8d0c65309c81 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:10.064809 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2ea8378-4987-439b-8bfc-8d0c65309c81 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:10.116935 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: WARNING keystone.common.rbac_enforcer.enforcer [None req-e2ea8378-4987-439b-8bfc-8d0c65309c81 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Sep 11 00:38:10.117283 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2ea8378-4987-439b-8bfc-8d0c65309c81 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:10.127061 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 2/4] 10.222.0.27 () {58 vars in 1324 bytes} [Sun Sep 11 00:38:09 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 292 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 11 00:38:10.132946 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f0e354d-044d-4bdc-8b6d-7300be0d214e None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:10.150483 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f0e354d-044d-4bdc-8b6d-7300be0d214e None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:10.155708 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2909ee42-5f39-4e30-9779-36023b4d944d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:10.156851 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2909ee42-5f39-4e30-9779-36023b4d944d admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:10.157193 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2909ee42-5f39-4e30-9779-36023b4d944d admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:10.157473 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2909ee42-5f39-4e30-9779-36023b4d944d admin admin] PATH_INFO: `/v3/projects/12356f1769944cc8be5c533d062adf65` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:10.159104 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2909ee42-5f39-4e30-9779-36023b4d944d admin admin] RBAC: Authorizing `identity:get_project(project_id=12356f1769944cc8be5c533d062adf65)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:10.211357 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: WARNING keystone.common.rbac_enforcer.enforcer [None req-2909ee42-5f39-4e30-9779-36023b4d944d admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Sep 11 00:38:10.211703 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2909ee42-5f39-4e30-9779-36023b4d944d admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:10.216865 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 3/5] 10.222.0.27 () {56 vars in 1207 bytes} [Sun Sep 11 00:38:10 2022] GET /identity/v3/projects/12356f1769944cc8be5c533d062adf65 => generated 344 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:12.357188 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2757d831-c2ca-4e5e-b615-d4f50045740d None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:12.357725 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2757d831-c2ca-4e5e-b615-d4f50045740d None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:12.357725 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2757d831-c2ca-4e5e-b615-d4f50045740d None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:12.361753 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 3/6] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:38:12 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 11 00:38:12.367231 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc0d385-f7ca-4252-961f-bf18a33aea47 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:12.367670 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc0d385-f7ca-4252-961f-bf18a33aea47 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:12.367765 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc0d385-f7ca-4252-961f-bf18a33aea47 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:12.389233 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-cfc0d385-f7ca-4252-961f-bf18a33aea47 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:38:12.390763 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-cfc0d385-f7ca-4252-961f-bf18a33aea47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:38:12.398551 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 4/7] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:38:12 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 384 bytes (2 switches on core 0) Sep 11 00:38:12.671062 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2757d831-c2ca-4e5e-b615-d4f50045740d None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:12.683153 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2757d831-c2ca-4e5e-b615-d4f50045740d None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:12.687369 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9d4273c-d9c0-4151-8928-96f6c4f63531 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:12.687962 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d4273c-d9c0-4151-8928-96f6c4f63531 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:12.688244 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d4273c-d9c0-4151-8928-96f6c4f63531 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:12.688516 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d4273c-d9c0-4151-8928-96f6c4f63531 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:12.696409 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9d4273c-d9c0-4151-8928-96f6c4f63531 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:12.698204 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9d4273c-d9c0-4151-8928-96f6c4f63531 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:12.706877 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 4/8] 10.222.0.27 () {58 vars in 1324 bytes} [Sun Sep 11 00:38:12 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 11 00:38:12.712155 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfc0d385-f7ca-4252-961f-bf18a33aea47 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:12.721792 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfc0d385-f7ca-4252-961f-bf18a33aea47 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:12.726040 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c10a358-43d1-46fe-b3b8-b7ecc8ccfada admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:12.726650 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c10a358-43d1-46fe-b3b8-b7ecc8ccfada admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:12.726962 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c10a358-43d1-46fe-b3b8-b7ecc8ccfada admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:12.727232 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c10a358-43d1-46fe-b3b8-b7ecc8ccfada admin admin] PATH_INFO: `/v3/users/36edaa883d654921ac17e7745c0ff320` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:12.728521 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c10a358-43d1-46fe-b3b8-b7ecc8ccfada admin admin] RBAC: Authorizing `identity:get_user(user_id=36edaa883d654921ac17e7745c0ff320)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:12.731216 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c10a358-43d1-46fe-b3b8-b7ecc8ccfada admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:12.732706 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 5/9] 10.222.0.27 () {56 vars in 1198 bytes} [Sun Sep 11 00:38:12 2022] GET /identity/v3/users/36edaa883d654921ac17e7745c0ff320 => generated 247 bytes in 21 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:14.940837 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331fe2f0-b1bf-462d-a936-d70c724d26c5 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:14.941360 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331fe2f0-b1bf-462d-a936-d70c724d26c5 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:14.941360 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331fe2f0-b1bf-462d-a936-d70c724d26c5 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:14.942691 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 5/10] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:38:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:38:14.948561 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee752a39-0ec0-498b-b25e-9e04200215aa None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:14.949011 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee752a39-0ec0-498b-b25e-9e04200215aa None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:14.949107 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee752a39-0ec0-498b-b25e-9e04200215aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:14.967358 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-ee752a39-0ec0-498b-b25e-9e04200215aa None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:38:14.968543 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-ee752a39-0ec0-498b-b25e-9e04200215aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:38:14.977137 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f62478-9910-4f3e-9193-29ab7768334a None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:14.977419 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f62478-9910-4f3e-9193-29ab7768334a None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:14.977691 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f62478-9910-4f3e-9193-29ab7768334a None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:14.979217 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 6/11] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:38:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 11 00:38:14.984488 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14aad8cb-1295-4e0c-9b64-18cc82c1bba9 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:14.984772 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14aad8cb-1295-4e0c-9b64-18cc82c1bba9 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:14.985046 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14aad8cb-1295-4e0c-9b64-18cc82c1bba9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:15.027057 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG passlib.handlers.bcrypt [None req-14aad8cb-1295-4e0c-9b64-18cc82c1bba9 None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=87924) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Sep 11 00:38:15.027439 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG passlib.handlers.bcrypt [None req-14aad8cb-1295-4e0c-9b64-18cc82c1bba9 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=87924) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Sep 11 00:38:15.052478 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 6/12] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:38:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:38:15.077838 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-14aad8cb-1295-4e0c-9b64-18cc82c1bba9 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:38:15.079486 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-14aad8cb-1295-4e0c-9b64-18cc82c1bba9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:38:15.179384 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 7/13] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:38:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:38:15.320556 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee752a39-0ec0-498b-b25e-9e04200215aa None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:15.344569 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:15.426907 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee752a39-0ec0-498b-b25e-9e04200215aa None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:15.456360 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14aad8cb-1295-4e0c-9b64-18cc82c1bba9 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:15.461248 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c421002-e2b8-4f0b-9dae-b83be830ae5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:15.461885 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c421002-e2b8-4f0b-9dae-b83be830ae5b None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:15.462165 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c421002-e2b8-4f0b-9dae-b83be830ae5b None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:15.462441 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c421002-e2b8-4f0b-9dae-b83be830ae5b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:15.474424 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c421002-e2b8-4f0b-9dae-b83be830ae5b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:38:15.484129 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-1c421002-e2b8-4f0b-9dae-b83be830ae5b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:38:15.484129 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:38:15.484129 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:38:15.484129 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:38:15.484129 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:38:15.484129 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:38:15.484129 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:38:15.484129 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:38:15.484129 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:38:15.484129 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:38:15.484129 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:38:15.484129 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:38:15.484129 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:38:15.484129 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:38:15.484129 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:38:15.484129 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:38:15.485199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:38:15.485199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:38:15.485199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:38:15.485199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:38:15.485199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:38:15.485199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:38:15.485199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:38:15.485199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:38:15.485199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:38:15.485199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:38:15.485199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:38:15.485199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:38:15.485199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:38:15.485199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:38:15.485199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:38:15.485199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:38:15.486064 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:38:15.486064 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:38:15.486064 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:38:15.486064 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:38:15.486064 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:38:15.486064 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:38:15.486064 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:38:15.486064 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:38:15.486064 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:38:15.486064 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:38:15.486064 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:38:15.486064 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:38:15.486064 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 7/14] 10.222.0.27 () {56 vars in 1096 bytes} [Sun Sep 11 00:38:15 2022] GET /identity/v3/roles/admin => generated 83 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:38:15.497441 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c421002-e2b8-4f0b-9dae-b83be830ae5b None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:15.559847 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14aad8cb-1295-4e0c-9b64-18cc82c1bba9 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:15.562150 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:15.580808 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:15.598317 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c6e3ff-eb37-4aa5-90a0-be7000692a0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:15.599224 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c6e3ff-eb37-4aa5-90a0-be7000692a0b None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:15.599509 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c6e3ff-eb37-4aa5-90a0-be7000692a0b None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:15.599782 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c6e3ff-eb37-4aa5-90a0-be7000692a0b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:15.599886 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c421002-e2b8-4f0b-9dae-b83be830ae5b None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:15.634168 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e729a7ee-8dcd-402b-9017-30db618bde54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:15.634766 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e729a7ee-8dcd-402b-9017-30db618bde54 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:15.635160 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e729a7ee-8dcd-402b-9017-30db618bde54 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:15.635429 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e729a7ee-8dcd-402b-9017-30db618bde54 None admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:15.636185 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e729a7ee-8dcd-402b-9017-30db618bde54 None admin] RBAC: Adding query filter params (name=admin) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:38:15.636487 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e729a7ee-8dcd-402b-9017-30db618bde54 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:15.637884 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e729a7ee-8dcd-402b-9017-30db618bde54 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:15.649495 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 8/15] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:38:15 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:38:15.655690 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e729a7ee-8dcd-402b-9017-30db618bde54 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:15.666500 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40c6e3ff-eb37-4aa5-90a0-be7000692a0b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:15.668120 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40c6e3ff-eb37-4aa5-90a0-be7000692a0b None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:15.757540 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e729a7ee-8dcd-402b-9017-30db618bde54 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:15.766218 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 8/16] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:38:15 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 11 00:38:15.771725 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c6e3ff-eb37-4aa5-90a0-be7000692a0b None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:15.774513 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:15.800367 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c52b1fca-269d-4294-bb19-80901927d85d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:15.800995 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52b1fca-269d-4294-bb19-80901927d85d None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:15.801277 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52b1fca-269d-4294-bb19-80901927d85d None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:15.801544 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52b1fca-269d-4294-bb19-80901927d85d None admin] PATH_INFO: `/v3/users/36edaa883d654921ac17e7745c0ff320` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:15.802700 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c52b1fca-269d-4294-bb19-80901927d85d None admin] RBAC: Authorizing `identity:get_user(user_id=36edaa883d654921ac17e7745c0ff320)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:15.804198 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c52b1fca-269d-4294-bb19-80901927d85d None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:15.805773 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 9/17] 10.222.0.27 () {56 vars in 1177 bytes} [Sun Sep 11 00:38:15 2022] GET /identity/v3/users/36edaa883d654921ac17e7745c0ff320 => generated 247 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:38:15.811161 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c52b1fca-269d-4294-bb19-80901927d85d None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:15.825178 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:15.872141 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c6e3ff-eb37-4aa5-90a0-be7000692a0b None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:15.905393 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e4ed5c0-75b4-455d-a653-e2aaabe48650 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:15.906234 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4ed5c0-75b4-455d-a653-e2aaabe48650 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:15.906510 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4ed5c0-75b4-455d-a653-e2aaabe48650 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:15.906774 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4ed5c0-75b4-455d-a653-e2aaabe48650 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:15.910125 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c52b1fca-269d-4294-bb19-80901927d85d None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:15.925428 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e4ed5c0-75b4-455d-a653-e2aaabe48650 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:15.926935 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e4ed5c0-75b4-455d-a653-e2aaabe48650 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:15.945073 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-6e4ed5c0-75b4-455d-a653-e2aaabe48650 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:38:15.945073 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:38:15.945073 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:38:15.945073 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:38:15.945073 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:38:15.945073 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:38:15.945073 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:38:15.945073 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:38:15.945073 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:38:15.945073 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:38:15.945073 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:38:15.945073 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:38:15.945073 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:38:15.945073 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:38:15.945073 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:38:15.945073 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:38:15.946420 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:38:15.946420 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:38:15.946420 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:38:15.946420 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:38:15.946420 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:38:15.946420 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:38:15.946420 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:38:15.946420 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:38:15.946420 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:38:15.946420 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:38:15.946420 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:38:15.946420 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:38:15.946420 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:38:15.946420 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:38:15.946420 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:38:15.946420 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:38:15.946420 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:38:15.947288 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:38:15.947288 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:38:15.947288 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:38:15.947478 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5894b6a6-3363-4396-be47-8fe484ded5bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:15.947583 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 9/18] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:38:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:38:15.947657 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5894b6a6-3363-4396-be47-8fe484ded5bf None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:15.947657 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5894b6a6-3363-4396-be47-8fe484ded5bf None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:15.947657 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5894b6a6-3363-4396-be47-8fe484ded5bf None admin] PATH_INFO: `/v3/domains/default` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:15.948489 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5894b6a6-3363-4396-be47-8fe484ded5bf None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:15.949925 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5894b6a6-3363-4396-be47-8fe484ded5bf None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:15.951478 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 10/19] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:38:15 2022] GET /identity/v3/domains/default => generated 199 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:38:15.952009 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e4ed5c0-75b4-455d-a653-e2aaabe48650 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:15.957055 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5894b6a6-3363-4396-be47-8fe484ded5bf None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:16.008319 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:16.036147 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:16.056897 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5894b6a6-3363-4396-be47-8fe484ded5bf None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:16.058300 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e4ed5c0-75b4-455d-a653-e2aaabe48650 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:16.090725 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c48d046c-542a-4a7f-ab12-18a720efc199 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:16.091351 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48d046c-542a-4a7f-ab12-18a720efc199 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:16.091628 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48d046c-542a-4a7f-ab12-18a720efc199 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:16.091897 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48d046c-542a-4a7f-ab12-18a720efc199 None admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:16.092219 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a88400-adf4-4017-abe8-4124665f5364 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:16.092537 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c48d046c-542a-4a7f-ab12-18a720efc199 None admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:38:16.092825 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a88400-adf4-4017-abe8-4124665f5364 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:16.092910 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c48d046c-542a-4a7f-ab12-18a720efc199 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:16.093103 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a88400-adf4-4017-abe8-4124665f5364 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:16.093372 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a88400-adf4-4017-abe8-4124665f5364 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:16.094097 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a88400-adf4-4017-abe8-4124665f5364 None admin] RBAC: Adding query filter params (role.id=7d9213c1187c4b5182bf8132cf23ea19, scope.domain.id=default, user.id=36edaa883d654921ac17e7745c0ff320) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:38:16.094252 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c48d046c-542a-4a7f-ab12-18a720efc199 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:16.094400 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a88400-adf4-4017-abe8-4124665f5364 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:16.095859 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a88400-adf4-4017-abe8-4124665f5364 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:16.112856 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 11/20] 10.222.0.27 () {56 vars in 1322 bytes} [Sun Sep 11 00:38:15 2022] GET /identity/v3/role_assignments?user.id=36edaa883d654921ac17e7745c0ff320&scope.domain.id=default&role.id=7d9213c1187c4b5182bf8132cf23ea19 => generated 233 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:38:16.113072 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 10/21] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:38:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:38:18.730342 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084f0f3a-72d0-46d9-b99b-86cfd7ed6c6d None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:18.731390 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084f0f3a-72d0-46d9-b99b-86cfd7ed6c6d None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:18.731871 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084f0f3a-72d0-46d9-b99b-86cfd7ed6c6d None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:18.733403 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 12/22] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:38:18 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:38:18.761342 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-708cb538-4b6f-45ae-9f8e-fa00ea8afe72 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:18.761873 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-708cb538-4b6f-45ae-9f8e-fa00ea8afe72 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:18.761973 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-708cb538-4b6f-45ae-9f8e-fa00ea8afe72 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:18.793010 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-708cb538-4b6f-45ae-9f8e-fa00ea8afe72 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:38:18.794216 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-708cb538-4b6f-45ae-9f8e-fa00ea8afe72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:38:18.817265 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9eb125c-9eb1-4c19-ba05-b157bdfe28e0 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:18.818107 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9eb125c-9eb1-4c19-ba05-b157bdfe28e0 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:18.818913 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9eb125c-9eb1-4c19-ba05-b157bdfe28e0 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:18.820450 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 13/23] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:38:18 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:38:18.826427 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6eb1be-6673-48b4-ae04-3127ce8c5096 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:18.827086 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6eb1be-6673-48b4-ae04-3127ce8c5096 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:18.827741 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6eb1be-6673-48b4-ae04-3127ce8c5096 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:18.860615 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-8b6eb1be-6673-48b4-ae04-3127ce8c5096 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:38:18.862219 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-8b6eb1be-6673-48b4-ae04-3127ce8c5096 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:38:18.933109 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 11/24] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:38:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:38:18.988379 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 14/25] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:38:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:38:19.109033 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a82d92f-7ba8-4635-ab93-89eb31076396 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:19.109322 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a82d92f-7ba8-4635-ab93-89eb31076396 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:19.109600 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a82d92f-7ba8-4635-ab93-89eb31076396 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:19.110999 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 12/26] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:38:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:38:19.117017 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-287c683e-12e3-42e8-ad0e-7064d8652c5b None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:19.117301 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-287c683e-12e3-42e8-ad0e-7064d8652c5b None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:19.117754 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-287c683e-12e3-42e8-ad0e-7064d8652c5b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:19.136636 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf288bd1-59a9-459c-bcd6-e0d42c90c535 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:19.136922 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf288bd1-59a9-459c-bcd6-e0d42c90c535 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:19.137196 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf288bd1-59a9-459c-bcd6-e0d42c90c535 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:19.138693 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 13/27] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:38:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:38:19.143733 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda260a5-a8ba-413f-88c1-8a4b308013a3 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:19.144193 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda260a5-a8ba-413f-88c1-8a4b308013a3 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:19.144618 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda260a5-a8ba-413f-88c1-8a4b308013a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:19.150810 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-287c683e-12e3-42e8-ad0e-7064d8652c5b None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:38:19.150810 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-287c683e-12e3-42e8-ad0e-7064d8652c5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:38:19.173390 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-fda260a5-a8ba-413f-88c1-8a4b308013a3 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:38:19.174817 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-fda260a5-a8ba-413f-88c1-8a4b308013a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:38:19.272592 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 15/28] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:38:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:38:19.274768 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:19.319997 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 14/29] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:38:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 11 00:38:19.332362 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-287c683e-12e3-42e8-ad0e-7064d8652c5b None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:19.340028 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:19.442116 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda260a5-a8ba-413f-88c1-8a4b308013a3 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:19.474626 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-287c683e-12e3-42e8-ad0e-7064d8652c5b None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:19.530905 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef6bd76-49ed-4d2a-9e27-8f233c605365 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:19.530905 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef6bd76-49ed-4d2a-9e27-8f233c605365 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:19.530905 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef6bd76-49ed-4d2a-9e27-8f233c605365 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:19.530905 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef6bd76-49ed-4d2a-9e27-8f233c605365 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:19.547740 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eef6bd76-49ed-4d2a-9e27-8f233c605365 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:38:19.553078 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-eef6bd76-49ed-4d2a-9e27-8f233c605365 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:38:19.553078 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:38:19.553078 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:38:19.553078 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:38:19.553078 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:38:19.553078 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:38:19.553078 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:38:19.553078 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:38:19.553078 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:38:19.553078 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:38:19.553078 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:38:19.553078 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:38:19.553078 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:38:19.553078 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:38:19.553078 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:38:19.553078 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:38:19.553920 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:38:19.553920 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:38:19.553920 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:38:19.553920 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:38:19.553920 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:38:19.553920 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:38:19.553920 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:38:19.553920 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:38:19.553920 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:38:19.553920 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:38:19.553920 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:38:19.553920 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:38:19.553920 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:38:19.553920 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:38:19.553920 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:38:19.553920 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:38:19.554697 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:38:19.554697 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:38:19.554697 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:38:19.554697 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:38:19.554697 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:38:19.554697 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:38:19.554697 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:38:19.554697 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:38:19.554697 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:38:19.554697 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:38:19.554697 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:38:19.554697 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:38:19.554697 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 16/30] 10.222.0.27 () {56 vars in 1096 bytes} [Sun Sep 11 00:38:19 2022] GET /identity/v3/roles/admin => generated 83 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:38:19.566834 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef6bd76-49ed-4d2a-9e27-8f233c605365 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:19.667099 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda260a5-a8ba-413f-88c1-8a4b308013a3 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:19.675478 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:19.736508 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef6bd76-49ed-4d2a-9e27-8f233c605365 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:19.739229 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e44afb9-b417-4725-ba51-0634373393f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:19.739854 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e44afb9-b417-4725-ba51-0634373393f2 None admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:19.740133 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e44afb9-b417-4725-ba51-0634373393f2 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:19.740400 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e44afb9-b417-4725-ba51-0634373393f2 None admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:19.741090 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e44afb9-b417-4725-ba51-0634373393f2 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:19.742498 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e44afb9-b417-4725-ba51-0634373393f2 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:19.782401 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8d618c-b7f2-4631-95c7-216ec87b32c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:19.783362 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8d618c-b7f2-4631-95c7-216ec87b32c5 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:19.783899 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8d618c-b7f2-4631-95c7-216ec87b32c5 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:19.784373 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8d618c-b7f2-4631-95c7-216ec87b32c5 None admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:19.785575 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f8d618c-b7f2-4631-95c7-216ec87b32c5 None admin] RBAC: Adding query filter params (name=admin) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:38:19.786077 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f8d618c-b7f2-4631-95c7-216ec87b32c5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:19.788288 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f8d618c-b7f2-4631-95c7-216ec87b32c5 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:19.798142 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 17/31] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:38:19 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:38:19.799306 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8d618c-b7f2-4631-95c7-216ec87b32c5 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:19.887073 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 15/32] 10.222.0.27 () {60 vars in 1131 bytes} [Sun Sep 11 00:38:19 2022] POST /identity/v3/roles => generated 219 bytes in 439 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:38:19.888232 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e44afb9-b417-4725-ba51-0634373393f2 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:19.910670 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8d618c-b7f2-4631-95c7-216ec87b32c5 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:19.954219 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:19.976056 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:19.994533 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d34a433-1f2b-4877-99a2-2be638f95704 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:20.001183 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d34a433-1f2b-4877-99a2-2be638f95704 None admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:20.001577 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d34a433-1f2b-4877-99a2-2be638f95704 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:20.001961 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d34a433-1f2b-4877-99a2-2be638f95704 None admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:20.002799 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d34a433-1f2b-4877-99a2-2be638f95704 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:20.004354 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d34a433-1f2b-4877-99a2-2be638f95704 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:20.040536 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e44afb9-b417-4725-ba51-0634373393f2 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:20.090652 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b5ac3f1-9e8c-4bc8-9e43-029105c99751 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:20.091789 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b5ac3f1-9e8c-4bc8-9e43-029105c99751 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:20.092398 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b5ac3f1-9e8c-4bc8-9e43-029105c99751 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:20.092878 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b5ac3f1-9e8c-4bc8-9e43-029105c99751 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:20.106635 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b5ac3f1-9e8c-4bc8-9e43-029105c99751 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:20.109466 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b5ac3f1-9e8c-4bc8-9e43-029105c99751 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:20.126605 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-8b5ac3f1-9e8c-4bc8-9e43-029105c99751 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:38:20.126605 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:38:20.126605 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:38:20.126605 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:38:20.126605 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:38:20.126605 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:38:20.126605 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:38:20.126605 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:38:20.126605 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:38:20.126605 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:38:20.126605 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:38:20.126605 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:38:20.126605 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:38:20.126605 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:38:20.126605 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:38:20.126605 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:38:20.127518 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:38:20.127518 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:38:20.127518 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:38:20.127518 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:38:20.127518 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:38:20.127518 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:38:20.127518 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:38:20.127518 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:38:20.127518 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:38:20.127518 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:38:20.127518 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:38:20.127518 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:38:20.127518 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:38:20.127518 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:38:20.127518 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:38:20.127518 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:38:20.127518 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:38:20.128355 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:38:20.128355 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:38:20.128355 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:38:20.130144 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 16/33] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:38:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:38:20.131192 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b5ac3f1-9e8c-4bc8-9e43-029105c99751 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:20.167772 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 18/34] 10.222.0.27 () {60 vars in 1131 bytes} [Sun Sep 11 00:38:19 2022] POST /identity/v3/roles => generated 225 bytes in 369 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:38:20.168672 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d34a433-1f2b-4877-99a2-2be638f95704 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:20.242238 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:20.257947 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b5ac3f1-9e8c-4bc8-9e43-029105c99751 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:20.295926 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94e238c-c334-4cf3-b849-b4800a0c0b43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:20.296653 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94e238c-c334-4cf3-b849-b4800a0c0b43 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:20.297035 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94e238c-c334-4cf3-b849-b4800a0c0b43 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:20.297408 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94e238c-c334-4cf3-b849-b4800a0c0b43 None admin] PATH_INFO: `/v3/users/36edaa883d654921ac17e7745c0ff320` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:20.299375 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d94e238c-c334-4cf3-b849-b4800a0c0b43 None admin] RBAC: Authorizing `identity:get_user(user_id=36edaa883d654921ac17e7745c0ff320)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:20.301018 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d94e238c-c334-4cf3-b849-b4800a0c0b43 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:20.303007 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 17/35] 10.222.0.27 () {56 vars in 1177 bytes} [Sun Sep 11 00:38:20 2022] GET /identity/v3/users/36edaa883d654921ac17e7745c0ff320 => generated 247 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:20.304180 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c03e126-3f0d-4634-ba8d-2b0f796442f9 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:20.304564 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c03e126-3f0d-4634-ba8d-2b0f796442f9 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:20.304937 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c03e126-3f0d-4634-ba8d-2b0f796442f9 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:20.306246 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 18/36] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:38:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:38:20.311842 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c03e126-3f0d-4634-ba8d-2b0f796442f9 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:20.343445 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d34a433-1f2b-4877-99a2-2be638f95704 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:20.391258 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1badbf4-e85c-41ca-8223-bb7374e6adfe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:20.392008 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1badbf4-e85c-41ca-8223-bb7374e6adfe None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:20.392297 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1badbf4-e85c-41ca-8223-bb7374e6adfe None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:20.392694 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1badbf4-e85c-41ca-8223-bb7374e6adfe None admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:20.393353 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1badbf4-e85c-41ca-8223-bb7374e6adfe None admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:38:20.393657 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1badbf4-e85c-41ca-8223-bb7374e6adfe None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:20.395244 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1badbf4-e85c-41ca-8223-bb7374e6adfe None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:20.414769 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 19/37] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:38:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:20.417706 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7578d6-f8c3-46b6-90da-7a60d48c0a2e None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:20.418155 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7578d6-f8c3-46b6-90da-7a60d48c0a2e None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:20.418859 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7578d6-f8c3-46b6-90da-7a60d48c0a2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:20.443376 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-3b7578d6-f8c3-46b6-90da-7a60d48c0a2e None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:38:20.445000 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-3b7578d6-f8c3-46b6-90da-7a60d48c0a2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:38:20.445904 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c03e126-3f0d-4634-ba8d-2b0f796442f9 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:20.485275 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08228994-04ac-4762-96d9-6854d31425b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:20.485921 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08228994-04ac-4762-96d9-6854d31425b4 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:20.486206 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08228994-04ac-4762-96d9-6854d31425b4 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:20.486482 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08228994-04ac-4762-96d9-6854d31425b4 None admin] PATH_INFO: `/v3/domains/default` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:20.488778 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08228994-04ac-4762-96d9-6854d31425b4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:20.489988 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08228994-04ac-4762-96d9-6854d31425b4 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:20.491643 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 19/38] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:38:20 2022] GET /identity/v3/domains/default => generated 199 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:20.492474 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08228994-04ac-4762-96d9-6854d31425b4 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:20.498650 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:20.523920 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:20.529950 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 20/39] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:38:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:38:20.539178 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b7578d6-f8c3-46b6-90da-7a60d48c0a2e None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:20.622342 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08228994-04ac-4762-96d9-6854d31425b4 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:20.666588 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e849ee99-d1cc-4aa9-87b6-e63e226ae4ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:20.667416 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e849ee99-d1cc-4aa9-87b6-e63e226ae4ee None admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:20.667792 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e849ee99-d1cc-4aa9-87b6-e63e226ae4ee None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:20.668163 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e849ee99-d1cc-4aa9-87b6-e63e226ae4ee None admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:20.668607 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b7578d6-f8c3-46b6-90da-7a60d48c0a2e None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:20.669033 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e849ee99-d1cc-4aa9-87b6-e63e226ae4ee None admin] RBAC: Authorizing `identity:create_project()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:20.671383 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e849ee99-d1cc-4aa9-87b6-e63e226ae4ee None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:20.710519 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 20/40] 10.222.0.27 () {60 vars in 1141 bytes} [Sun Sep 11 00:38:20 2022] POST /identity/v3/projects => generated 301 bytes in 218 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:38:20.782659 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7717a5ef-f5e4-465c-9ec0-e5ca8c9db037 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:20.782981 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7717a5ef-f5e4-465c-9ec0-e5ca8c9db037 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:20.783266 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7717a5ef-f5e4-465c-9ec0-e5ca8c9db037 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:20.785211 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 21/41] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:38:20 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 11 00:38:20.791970 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd57a180-fe6a-4419-ac73-6485841a1de3 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:20.792752 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd57a180-fe6a-4419-ac73-6485841a1de3 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:20.793238 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd57a180-fe6a-4419-ac73-6485841a1de3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:20.817539 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-cd57a180-fe6a-4419-ac73-6485841a1de3 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:38:20.819520 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-cd57a180-fe6a-4419-ac73-6485841a1de3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:38:20.830649 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b813a20a-15ac-4acc-bd11-8a9fb1ddbf5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:20.831395 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b813a20a-15ac-4acc-bd11-8a9fb1ddbf5c None admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:20.831674 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b813a20a-15ac-4acc-bd11-8a9fb1ddbf5c None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:20.831952 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b813a20a-15ac-4acc-bd11-8a9fb1ddbf5c None admin] PATH_INFO: `/v3/domains/default/users/36edaa883d654921ac17e7745c0ff320/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:20.833981 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b813a20a-15ac-4acc-bd11-8a9fb1ddbf5c None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=36edaa883d654921ac17e7745c0ff320, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:20.835460 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b813a20a-15ac-4acc-bd11-8a9fb1ddbf5c None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:20.855185 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 21/42] 10.222.0.27 () {58 vars in 1361 bytes} [Sun Sep 11 00:38:20 2022] PUT /identity/v3/domains/default/users/36edaa883d654921ac17e7745c0ff320/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 325 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:38:20.856433 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b813a20a-15ac-4acc-bd11-8a9fb1ddbf5c None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:20.905033 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:20.936873 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 22/43] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:38:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 11 00:38:20.999167 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b813a20a-15ac-4acc-bd11-8a9fb1ddbf5c None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:21.042599 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9aac02-5042-42be-b686-727915d53c37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:21.043357 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9aac02-5042-42be-b686-727915d53c37 None admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:21.043640 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9aac02-5042-42be-b686-727915d53c37 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:21.043915 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9aac02-5042-42be-b686-727915d53c37 None admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:21.044606 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af9aac02-5042-42be-b686-727915d53c37 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:21.046019 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af9aac02-5042-42be-b686-727915d53c37 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:21.062954 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 22/44] 10.222.0.27 () {60 vars in 1131 bytes} [Sun Sep 11 00:38:20 2022] POST /identity/v3/roles => generated 223 bytes in 207 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:38:21.218113 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd57a180-fe6a-4419-ac73-6485841a1de3 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:21.226484 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:21.324985 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd57a180-fe6a-4419-ac73-6485841a1de3 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:21.359175 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75aeee59-7cd1-46e2-8eec-0da5bdf5491a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:21.359812 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75aeee59-7cd1-46e2-8eec-0da5bdf5491a None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:21.360094 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75aeee59-7cd1-46e2-8eec-0da5bdf5491a None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:21.360367 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75aeee59-7cd1-46e2-8eec-0da5bdf5491a None admin] PATH_INFO: `/v3/domains/default` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:21.361477 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75aeee59-7cd1-46e2-8eec-0da5bdf5491a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:21.362968 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75aeee59-7cd1-46e2-8eec-0da5bdf5491a None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:21.364543 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 23/45] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:38:21 2022] GET /identity/v3/domains/default => generated 199 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:21.371061 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9aac02-5042-42be-b686-727915d53c37 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:21.465645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:21.472595 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9aac02-5042-42be-b686-727915d53c37 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:21.506328 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc68eb38-f59f-4c24-84f4-9cf8edac93e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:21.506966 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc68eb38-f59f-4c24-84f4-9cf8edac93e6 None admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:21.507252 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc68eb38-f59f-4c24-84f4-9cf8edac93e6 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:21.507528 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc68eb38-f59f-4c24-84f4-9cf8edac93e6 None admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:21.508178 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc68eb38-f59f-4c24-84f4-9cf8edac93e6 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:21.509635 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc68eb38-f59f-4c24-84f4-9cf8edac93e6 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:21.538711 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 23/46] 10.222.0.27 () {60 vars in 1141 bytes} [Sun Sep 11 00:38:21 2022] POST /identity/v3/projects => generated 312 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:38:23.058011 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997e3871-da48-4c99-85c5-1687b44e7cfc None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:23.058568 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997e3871-da48-4c99-85c5-1687b44e7cfc None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:23.058568 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997e3871-da48-4c99-85c5-1687b44e7cfc None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:23.059936 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 24/47] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:38:23 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:38:23.065219 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28560aa2-5e0d-4603-8820-0b4c6f7c752c None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:23.065605 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28560aa2-5e0d-4603-8820-0b4c6f7c752c None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:23.065897 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28560aa2-5e0d-4603-8820-0b4c6f7c752c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:23.084857 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-28560aa2-5e0d-4603-8820-0b4c6f7c752c None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:38:23.086041 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-28560aa2-5e0d-4603-8820-0b4c6f7c752c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:38:23.152794 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 24/48] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:38:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 11 00:38:23.421757 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-997e3871-da48-4c99-85c5-1687b44e7cfc None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:23.505844 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:23.534499 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-997e3871-da48-4c99-85c5-1687b44e7cfc None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:23.569669 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dceb8def-7ace-4ffb-922d-8d19735fe9c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:23.570273 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dceb8def-7ace-4ffb-922d-8d19735fe9c2 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:23.570552 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dceb8def-7ace-4ffb-922d-8d19735fe9c2 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:23.570848 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dceb8def-7ace-4ffb-922d-8d19735fe9c2 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:23.583574 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dceb8def-7ace-4ffb-922d-8d19735fe9c2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:38:23.590727 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-dceb8def-7ace-4ffb-922d-8d19735fe9c2 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:38:23.590727 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:38:23.590727 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:38:23.590727 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:38:23.590727 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:38:23.590727 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:38:23.590727 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:38:23.590727 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:38:23.590727 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:38:23.590727 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:38:23.590727 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:38:23.590727 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:38:23.590727 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:38:23.590727 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:38:23.590727 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:38:23.590727 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:38:23.591642 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:38:23.591642 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:38:23.591642 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:38:23.591642 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:38:23.591642 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:38:23.591642 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:38:23.591642 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:38:23.591642 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:38:23.591642 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:38:23.591642 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:38:23.591642 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:38:23.591642 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:38:23.591642 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:38:23.591642 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:38:23.591642 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:38:23.591642 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:38:23.592682 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:38:23.592682 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:38:23.592682 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:38:23.592682 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:38:23.592682 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:38:23.592682 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:38:23.592682 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:38:23.592682 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:38:23.592682 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:38:23.592682 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:38:23.592682 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:38:23.592682 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:38:23.592682 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 25/49] 10.222.0.27 () {56 vars in 1096 bytes} [Sun Sep 11 00:38:23 2022] GET /identity/v3/roles/admin => generated 83 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:38:23.598479 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28560aa2-5e0d-4603-8820-0b4c6f7c752c None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:23.685569 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:23.709977 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28560aa2-5e0d-4603-8820-0b4c6f7c752c None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:23.733536 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7facded-ddd5-4c9b-9e46-68a1a833ffd5 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:23.733829 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7facded-ddd5-4c9b-9e46-68a1a833ffd5 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:23.734108 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7facded-ddd5-4c9b-9e46-68a1a833ffd5 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:23.735589 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 26/50] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:38:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:38:23.741108 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2186f47e-41ad-4028-956b-ec8a356f1570 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:23.741410 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2186f47e-41ad-4028-956b-ec8a356f1570 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:23.741683 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2186f47e-41ad-4028-956b-ec8a356f1570 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:23.745840 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46ed8a69-5ff0-4729-b427-532e138241d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:23.746452 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ed8a69-5ff0-4729-b427-532e138241d3 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:23.746727 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ed8a69-5ff0-4729-b427-532e138241d3 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:23.747689 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ed8a69-5ff0-4729-b427-532e138241d3 None admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:23.748376 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46ed8a69-5ff0-4729-b427-532e138241d3 None admin] RBAC: Adding query filter params (name=admin) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:38:23.748682 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46ed8a69-5ff0-4729-b427-532e138241d3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:23.750082 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46ed8a69-5ff0-4729-b427-532e138241d3 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:23.758601 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 25/51] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:38:23 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:23.763108 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-2186f47e-41ad-4028-956b-ec8a356f1570 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:38:23.764332 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-2186f47e-41ad-4028-956b-ec8a356f1570 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:38:23.764809 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46ed8a69-5ff0-4729-b427-532e138241d3 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:23.856622 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 27/52] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:38:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 11 00:38:23.887545 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46ed8a69-5ff0-4729-b427-532e138241d3 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:23.922202 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1255da9-6668-4c18-8f6c-2fa9070c816a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:23.922865 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1255da9-6668-4c18-8f6c-2fa9070c816a None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:23.923163 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1255da9-6668-4c18-8f6c-2fa9070c816a None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:23.923432 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1255da9-6668-4c18-8f6c-2fa9070c816a None admin] PATH_INFO: `/v3/users/36edaa883d654921ac17e7745c0ff320` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:23.924572 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1255da9-6668-4c18-8f6c-2fa9070c816a None admin] RBAC: Authorizing `identity:get_user(user_id=36edaa883d654921ac17e7745c0ff320)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:23.926019 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1255da9-6668-4c18-8f6c-2fa9070c816a None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:23.927843 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 26/53] 10.222.0.27 () {56 vars in 1177 bytes} [Sun Sep 11 00:38:23 2022] GET /identity/v3/users/36edaa883d654921ac17e7745c0ff320 => generated 247 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:38:23.933213 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2186f47e-41ad-4028-956b-ec8a356f1570 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:23.995147 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:24.032985 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2186f47e-41ad-4028-956b-ec8a356f1570 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:24.067472 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb236932-9466-4cfa-ab8a-36064731b916 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:24.068103 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb236932-9466-4cfa-ab8a-36064731b916 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:24.068370 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb236932-9466-4cfa-ab8a-36064731b916 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:24.068637 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb236932-9466-4cfa-ab8a-36064731b916 None admin] PATH_INFO: `/v3/domains/default` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:24.069743 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb236932-9466-4cfa-ab8a-36064731b916 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:24.071226 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb236932-9466-4cfa-ab8a-36064731b916 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:24.073123 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 28/54] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:38:23 2022] GET /identity/v3/domains/default => generated 199 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:38:24.078917 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1255da9-6668-4c18-8f6c-2fa9070c816a None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:24.091596 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:24.184027 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1255da9-6668-4c18-8f6c-2fa9070c816a None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:24.195194 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb236932-9466-4cfa-ab8a-36064731b916 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:24.217985 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f7e0db-e9d7-40cc-9f91-643fa7d2fb9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:24.218607 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f7e0db-e9d7-40cc-9f91-643fa7d2fb9a None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:24.218900 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f7e0db-e9d7-40cc-9f91-643fa7d2fb9a None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:24.219174 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f7e0db-e9d7-40cc-9f91-643fa7d2fb9a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:24.219897 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8f7e0db-e9d7-40cc-9f91-643fa7d2fb9a None admin] RBAC: Adding query filter params (role.id=7d9213c1187c4b5182bf8132cf23ea19, scope.domain.id=default, user.id=36edaa883d654921ac17e7745c0ff320) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:38:24.220204 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8f7e0db-e9d7-40cc-9f91-643fa7d2fb9a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:24.221642 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8f7e0db-e9d7-40cc-9f91-643fa7d2fb9a None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:24.227617 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 27/55] 10.222.0.27 () {56 vars in 1322 bytes} [Sun Sep 11 00:38:24 2022] GET /identity/v3/role_assignments?user.id=36edaa883d654921ac17e7745c0ff320&scope.domain.id=default&role.id=7d9213c1187c4b5182bf8132cf23ea19 => generated 530 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:24.300963 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb236932-9466-4cfa-ab8a-36064731b916 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:24.334372 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:24.340549 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0543cc87-77f2-4528-a884-8caff12aa6d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:24.341235 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0543cc87-77f2-4528-a884-8caff12aa6d8 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:24.341513 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0543cc87-77f2-4528-a884-8caff12aa6d8 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:24.341781 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0543cc87-77f2-4528-a884-8caff12aa6d8 None admin] PATH_INFO: `/v3/domains/default` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:24.342932 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0543cc87-77f2-4528-a884-8caff12aa6d8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:24.344398 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0543cc87-77f2-4528-a884-8caff12aa6d8 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:24.345977 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 29/56] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:38:24 2022] GET /identity/v3/domains/default => generated 199 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:38:24.351944 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f7e0db-e9d7-40cc-9f91-643fa7d2fb9a None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:24.427684 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:24.460168 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f7e0db-e9d7-40cc-9f91-643fa7d2fb9a None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:24.494392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6b2b1b6-32e0-4b10-b5a2-0b6260394297 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:24.495026 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b2b1b6-32e0-4b10-b5a2-0b6260394297 None admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:24.495312 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b2b1b6-32e0-4b10-b5a2-0b6260394297 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:24.495584 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b2b1b6-32e0-4b10-b5a2-0b6260394297 None admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:24.496261 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6b2b1b6-32e0-4b10-b5a2-0b6260394297 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:24.497711 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6b2b1b6-32e0-4b10-b5a2-0b6260394297 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:24.524285 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 28/57] 10.222.0.27 () {60 vars in 1140 bytes} [Sun Sep 11 00:38:24 2022] POST /identity/v3/projects => generated 298 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:38:26.724874 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd7c41d-20d3-437c-90df-79878e52b96e None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:26.725399 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd7c41d-20d3-437c-90df-79878e52b96e None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:26.725399 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd7c41d-20d3-437c-90df-79878e52b96e None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:26.726724 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 30/58] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:38:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:38:26.732569 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7674dd7-2463-4a09-9d27-fd905a3d789f None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:26.732977 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7674dd7-2463-4a09-9d27-fd905a3d789f None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:26.733111 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7674dd7-2463-4a09-9d27-fd905a3d789f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:26.751590 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-a7674dd7-2463-4a09-9d27-fd905a3d789f None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:38:26.752763 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-a7674dd7-2463-4a09-9d27-fd905a3d789f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:38:26.817767 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 29/59] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:38:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:38:27.089204 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd7c41d-20d3-437c-90df-79878e52b96e None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:27.290678 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd7c41d-20d3-437c-90df-79878e52b96e None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:27.323094 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b60b3681-d79e-4910-980e-e6a44b336ebc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:27.323860 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60b3681-d79e-4910-980e-e6a44b336ebc None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:27.324144 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60b3681-d79e-4910-980e-e6a44b336ebc None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:27.324417 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60b3681-d79e-4910-980e-e6a44b336ebc None admin] PATH_INFO: `/v3/domains/default` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:27.325548 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b60b3681-d79e-4910-980e-e6a44b336ebc None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:27.327003 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b60b3681-d79e-4910-980e-e6a44b336ebc None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:27.328549 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 31/60] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:38:27 2022] GET /identity/v3/domains/default => generated 199 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:27.334139 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7674dd7-2463-4a09-9d27-fd905a3d789f None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:27.388037 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:27.432400 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7674dd7-2463-4a09-9d27-fd905a3d789f None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:27.466131 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b53573c-5c94-4bf2-80e6-6438045bb901 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:27.466732 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b53573c-5c94-4bf2-80e6-6438045bb901 None admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:27.467038 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b53573c-5c94-4bf2-80e6-6438045bb901 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:27.467311 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b53573c-5c94-4bf2-80e6-6438045bb901 None admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:27.467968 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b53573c-5c94-4bf2-80e6-6438045bb901 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:27.469422 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b53573c-5c94-4bf2-80e6-6438045bb901 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:27.495445 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 30/61] 10.222.0.27 () {60 vars in 1132 bytes} [Sun Sep 11 00:38:27 2022] POST /identity/v3/users => generated 275 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:38:30.979756 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f816769-861e-4bd6-a6ad-2f2e8334a184 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:30.980282 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f816769-861e-4bd6-a6ad-2f2e8334a184 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:30.980282 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f816769-861e-4bd6-a6ad-2f2e8334a184 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:30.981653 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 32/62] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:38:30 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:38:30.987546 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03924c11-cec6-417e-a217-e3d3f1b5f483 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:30.988494 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03924c11-cec6-417e-a217-e3d3f1b5f483 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:30.988972 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03924c11-cec6-417e-a217-e3d3f1b5f483 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:30.996185 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc16919f-1330-4249-a8f6-c4c3b6c2fd88 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:30.996846 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc16919f-1330-4249-a8f6-c4c3b6c2fd88 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:30.997325 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc16919f-1330-4249-a8f6-c4c3b6c2fd88 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:30.998987 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 33/63] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:38:30 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:38:31.005178 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62369e6a-5137-4c87-acc1-9561c7e291d7 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:31.005802 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62369e6a-5137-4c87-acc1-9561c7e291d7 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:31.006313 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62369e6a-5137-4c87-acc1-9561c7e291d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:31.025099 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-03924c11-cec6-417e-a217-e3d3f1b5f483 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:38:31.027572 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-03924c11-cec6-417e-a217-e3d3f1b5f483 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:38:31.044218 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-62369e6a-5137-4c87-acc1-9561c7e291d7 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:38:31.045430 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-62369e6a-5137-4c87-acc1-9561c7e291d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:38:31.154108 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 31/64] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:38:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:38:31.155760 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6216a75f-72ed-4a7f-b34b-d76031c29676 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:31.156190 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6216a75f-72ed-4a7f-b34b-d76031c29676 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:31.156595 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6216a75f-72ed-4a7f-b34b-d76031c29676 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:31.158021 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 32/65] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:38:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:38:31.159224 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04da410-9ebb-43d4-8933-ba3b020ca0a1 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:31.160277 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04da410-9ebb-43d4-8933-ba3b020ca0a1 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:31.160732 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04da410-9ebb-43d4-8933-ba3b020ca0a1 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:31.162298 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 33/66] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:38:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:38:31.163480 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e85412-4984-42c8-8e0f-5e8898685fd1 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:31.163911 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e85412-4984-42c8-8e0f-5e8898685fd1 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:31.164334 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e85412-4984-42c8-8e0f-5e8898685fd1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:31.172506 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:31.178725 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 34/67] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:38:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 11 00:38:31.186982 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-e4e85412-4984-42c8-8e0f-5e8898685fd1 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:38:31.186982 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-e4e85412-4984-42c8-8e0f-5e8898685fd1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:38:31.188681 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fba612a-7a84-4551-b64a-3be3328d4ec9 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:31.189186 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fba612a-7a84-4551-b64a-3be3328d4ec9 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:31.189473 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fba612a-7a84-4551-b64a-3be3328d4ec9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:31.216780 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-6fba612a-7a84-4551-b64a-3be3328d4ec9 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:38:31.217984 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-6fba612a-7a84-4551-b64a-3be3328d4ec9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:38:31.234922 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:31.270707 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 34/68] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:38:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:38:31.318306 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 35/69] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:38:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:38:31.454973 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4e85412-4984-42c8-8e0f-5e8898685fd1 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:31.529457 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ea3fea-47fd-40ef-8bf2-a8a3fe85ffd2 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:31.530105 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ea3fea-47fd-40ef-8bf2-a8a3fe85ffd2 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:31.530585 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ea3fea-47fd-40ef-8bf2-a8a3fe85ffd2 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:31.532155 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 36/70] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:38:31 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:38:31.537587 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6181fa79-d2b5-4369-80dc-f3dc095c2a39 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:31.537952 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6181fa79-d2b5-4369-80dc-f3dc095c2a39 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:31.538303 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6181fa79-d2b5-4369-80dc-f3dc095c2a39 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:31.568172 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-6181fa79-d2b5-4369-80dc-f3dc095c2a39 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:38:31.569691 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-6181fa79-d2b5-4369-80dc-f3dc095c2a39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:38:31.585192 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4e85412-4984-42c8-8e0f-5e8898685fd1 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:31.635709 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b3eac9b-d4ce-4f33-9fed-c0f802a1d99d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:31.637206 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b3eac9b-d4ce-4f33-9fed-c0f802a1d99d None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:31.638051 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b3eac9b-d4ce-4f33-9fed-c0f802a1d99d None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:31.638823 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b3eac9b-d4ce-4f33-9fed-c0f802a1d99d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:31.647899 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 37/71] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:38:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 11 00:38:31.652195 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b3eac9b-d4ce-4f33-9fed-c0f802a1d99d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:38:31.655668 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6181fa79-d2b5-4369-80dc-f3dc095c2a39 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:31.657849 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-0b3eac9b-d4ce-4f33-9fed-c0f802a1d99d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:38:31.657849 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:38:31.657849 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:38:31.657849 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:38:31.657849 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:38:31.657849 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:38:31.657849 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:38:31.657849 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:38:31.657849 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:38:31.657849 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:38:31.657849 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:38:31.657849 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:38:31.657849 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:38:31.657849 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:38:31.657849 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:38:31.657849 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:38:31.658706 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:38:31.658706 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:38:31.658706 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:38:31.658706 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:38:31.658706 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:38:31.658706 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:38:31.658706 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:38:31.658706 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:38:31.658706 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:38:31.658706 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:38:31.658706 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:38:31.658706 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:38:31.658706 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:38:31.658706 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:38:31.658706 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:38:31.658706 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:38:31.659514 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:38:31.659514 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:38:31.659514 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:38:31.659514 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:38:31.659514 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:38:31.659514 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:38:31.659514 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:38:31.659514 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:38:31.659514 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:38:31.659514 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:38:31.659514 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:38:31.659514 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:38:31.659514 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 35/72] 10.222.0.27 () {56 vars in 1096 bytes} [Sun Sep 11 00:38:31 2022] GET /identity/v3/roles/admin => generated 83 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:38:31.660408 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b3eac9b-d4ce-4f33-9fed-c0f802a1d99d None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:31.697046 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:31.725154 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:31.771733 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6181fa79-d2b5-4369-80dc-f3dc095c2a39 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:31.772548 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b3eac9b-d4ce-4f33-9fed-c0f802a1d99d None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:31.807676 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d67cbcda-9634-4489-bbb0-496910e881d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:31.808187 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67cbcda-9634-4489-bbb0-496910e881d4 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:31.808463 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67cbcda-9634-4489-bbb0-496910e881d4 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:31.808730 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67cbcda-9634-4489-bbb0-496910e881d4 None admin] PATH_INFO: `/v3/roles/member` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:31.811456 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6aa0c1b-de7e-4e27-b277-c117eefef369 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:31.812085 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6aa0c1b-de7e-4e27-b277-c117eefef369 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:31.812369 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6aa0c1b-de7e-4e27-b277-c117eefef369 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:31.812640 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6aa0c1b-de7e-4e27-b277-c117eefef369 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:31.818550 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d67cbcda-9634-4489-bbb0-496910e881d4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:38:31.823267 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6aa0c1b-de7e-4e27-b277-c117eefef369 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:38:31.823772 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-d67cbcda-9634-4489-bbb0-496910e881d4 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 11 00:38:31.823772 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:38:31.823772 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:38:31.823772 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:38:31.823772 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:38:31.823772 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:38:31.823772 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:38:31.823772 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:38:31.823772 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:38:31.823772 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:38:31.823772 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:38:31.823772 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:38:31.823772 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:38:31.823772 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:38:31.823772 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:38:31.823772 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:38:31.824610 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:38:31.824610 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:38:31.824610 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:38:31.824610 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:38:31.824610 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:38:31.824610 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:38:31.824610 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:38:31.824610 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:38:31.824610 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:38:31.824610 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:38:31.824610 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:38:31.824610 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:38:31.824610 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:38:31.824610 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:38:31.824610 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:38:31.824610 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:38:31.825363 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:38:31.825363 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:38:31.825363 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:38:31.825363 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:38:31.825363 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:38:31.825363 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:38:31.825363 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:38:31.825363 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:38:31.825363 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:38:31.825363 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:38:31.825363 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 11 00:38:31.825363 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:38:31.825363 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 38/73] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:38:31 2022] GET /identity/v3/roles/member => generated 84 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:38:31.826067 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d67cbcda-9634-4489-bbb0-496910e881d4 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:31.828508 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-e6aa0c1b-de7e-4e27-b277-c117eefef369 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 11 00:38:31.828508 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:38:31.828508 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:38:31.828508 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:38:31.828508 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:38:31.828508 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:38:31.828508 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:38:31.828508 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:38:31.828508 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:38:31.828508 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:38:31.828508 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:38:31.828508 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:38:31.828508 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:38:31.828508 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:38:31.828508 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:38:31.828508 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:38:31.829365 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:38:31.829365 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:38:31.829365 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:38:31.829365 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:38:31.829365 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:38:31.829365 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:38:31.829365 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:38:31.829365 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:38:31.829365 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:38:31.829365 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:38:31.829365 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:38:31.829365 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:38:31.829365 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:38:31.829365 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:38:31.829365 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:38:31.829365 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:38:31.835746 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:38:31.835746 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:38:31.835746 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:38:31.835746 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:38:31.835746 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:38:31.835746 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:38:31.835746 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:38:31.835746 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:38:31.835746 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:38:31.835746 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:38:31.835746 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 11 00:38:31.835746 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:38:31.835746 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 36/74] 10.222.0.27 () {56 vars in 1114 bytes} [Sun Sep 11 00:38:31 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:38:31.835746 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6aa0c1b-de7e-4e27-b277-c117eefef369 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:31.930437 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:31.939334 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d67cbcda-9634-4489-bbb0-496910e881d4 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:31.940212 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6aa0c1b-de7e-4e27-b277-c117eefef369 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:31.957050 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:31.974601 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-497a9994-e9a8-4224-9f87-cac0bfb27564 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:31.974805 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3815421e-c97b-4546-931b-de03d2037e18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:31.975252 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497a9994-e9a8-4224-9f87-cac0bfb27564 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:31.975436 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3815421e-c97b-4546-931b-de03d2037e18 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:31.975525 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497a9994-e9a8-4224-9f87-cac0bfb27564 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:31.975715 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3815421e-c97b-4546-931b-de03d2037e18 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:31.975806 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497a9994-e9a8-4224-9f87-cac0bfb27564 None admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:31.975985 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3815421e-c97b-4546-931b-de03d2037e18 None admin] PATH_INFO: `/v3/domains/default` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:31.976451 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-497a9994-e9a8-4224-9f87-cac0bfb27564 None admin] RBAC: Adding query filter params (name=admin) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:38:31.976758 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-497a9994-e9a8-4224-9f87-cac0bfb27564 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:31.977099 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3815421e-c97b-4546-931b-de03d2037e18 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:31.978136 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-497a9994-e9a8-4224-9f87-cac0bfb27564 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:31.978625 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3815421e-c97b-4546-931b-de03d2037e18 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:31.980637 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 37/75] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:38:31 2022] GET /identity/v3/domains/default => generated 199 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:31.981164 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3815421e-c97b-4546-931b-de03d2037e18 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:31.990165 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 39/76] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:38:31 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:31.991434 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-497a9994-e9a8-4224-9f87-cac0bfb27564 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:32.084372 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3815421e-c97b-4546-931b-de03d2037e18 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:32.097074 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-497a9994-e9a8-4224-9f87-cac0bfb27564 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:32.120704 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f183fb-b895-49aa-bf94-8cd4d1ba8ee9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:32.121616 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f183fb-b895-49aa-bf94-8cd4d1ba8ee9 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:32.121893 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f183fb-b895-49aa-bf94-8cd4d1ba8ee9 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:32.122161 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f183fb-b895-49aa-bf94-8cd4d1ba8ee9 None admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:32.122849 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60f183fb-b895-49aa-bf94-8cd4d1ba8ee9 None admin] RBAC: Adding query filter params (name=member) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:38:32.123151 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60f183fb-b895-49aa-bf94-8cd4d1ba8ee9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:32.124530 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60f183fb-b895-49aa-bf94-8cd4d1ba8ee9 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:32.132349 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 38/77] 10.222.0.27 () {56 vars in 1101 bytes} [Sun Sep 11 00:38:31 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:32.133123 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f183fb-b895-49aa-bf94-8cd4d1ba8ee9 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:32.134046 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3b454a-ce6c-484d-8d8b-f9985c97ecac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:32.135320 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3b454a-ce6c-484d-8d8b-f9985c97ecac None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:32.135675 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3b454a-ce6c-484d-8d8b-f9985c97ecac None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:32.135956 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3b454a-ce6c-484d-8d8b-f9985c97ecac None admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:32.136624 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a3b454a-ce6c-484d-8d8b-f9985c97ecac None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:38:32.136929 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a3b454a-ce6c-484d-8d8b-f9985c97ecac None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:32.138309 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a3b454a-ce6c-484d-8d8b-f9985c97ecac None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:32.146339 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 40/78] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:38:31 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:32.148201 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3b454a-ce6c-484d-8d8b-f9985c97ecac None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:32.161262 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:32.191947 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:32.237838 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f183fb-b895-49aa-bf94-8cd4d1ba8ee9 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:32.248577 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3b454a-ce6c-484d-8d8b-f9985c97ecac None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:32.270798 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da13e5f9-83d9-4065-9dc7-8d8e7e85e0cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:32.271448 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da13e5f9-83d9-4065-9dc7-8d8e7e85e0cb None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:32.271729 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da13e5f9-83d9-4065-9dc7-8d8e7e85e0cb None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:32.271999 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da13e5f9-83d9-4065-9dc7-8d8e7e85e0cb None admin] PATH_INFO: `/v3/roles/member` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:32.282097 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2db45d-d243-4a5c-90c6-456c18626d8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:32.282337 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da13e5f9-83d9-4065-9dc7-8d8e7e85e0cb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:38:32.282689 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2db45d-d243-4a5c-90c6-456c18626d8a None admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:32.282995 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2db45d-d243-4a5c-90c6-456c18626d8a None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:32.283264 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2db45d-d243-4a5c-90c6-456c18626d8a None admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:32.283912 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb2db45d-d243-4a5c-90c6-456c18626d8a None admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:32.285868 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb2db45d-d243-4a5c-90c6-456c18626d8a None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:32.288088 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-da13e5f9-83d9-4065-9dc7-8d8e7e85e0cb None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 11 00:38:32.288088 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:38:32.288088 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:38:32.288088 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:38:32.288088 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:38:32.288088 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:38:32.288088 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:38:32.288088 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:38:32.288088 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:38:32.288088 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:38:32.288088 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:38:32.288088 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:38:32.288088 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:38:32.288088 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:38:32.288088 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:38:32.288088 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:38:32.289084 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:38:32.289084 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:38:32.289084 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:38:32.289084 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:38:32.289084 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:38:32.289084 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:38:32.289084 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:38:32.289084 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:38:32.289084 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:38:32.289084 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:38:32.289084 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:38:32.289084 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:38:32.289084 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:38:32.289084 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:38:32.289084 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:38:32.289084 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:38:32.290029 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:38:32.290029 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:38:32.290029 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:38:32.290029 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:38:32.290029 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:38:32.290029 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:38:32.290029 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:38:32.290029 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:38:32.290029 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:38:32.290029 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:38:32.290029 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 11 00:38:32.290029 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:38:32.290029 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 39/79] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:38:32 2022] GET /identity/v3/roles/member => generated 84 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:38:32.290029 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da13e5f9-83d9-4065-9dc7-8d8e7e85e0cb None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:32.315246 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 41/80] 10.222.0.27 () {60 vars in 1132 bytes} [Sun Sep 11 00:38:32 2022] POST /identity/v3/users => generated 289 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:38:32.316076 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2db45d-d243-4a5c-90c6-456c18626d8a None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:32.387404 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:32.411338 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da13e5f9-83d9-4065-9dc7-8d8e7e85e0cb None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:32.439122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:32.444963 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2db45d-d243-4a5c-90c6-456c18626d8a None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:32.450196 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb25fe4c-923c-4e3c-8f83-fa9e6075e259 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:32.450881 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb25fe4c-923c-4e3c-8f83-fa9e6075e259 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:32.451172 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb25fe4c-923c-4e3c-8f83-fa9e6075e259 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:32.451442 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb25fe4c-923c-4e3c-8f83-fa9e6075e259 None admin] PATH_INFO: `/v3/users/36edaa883d654921ac17e7745c0ff320` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:32.452654 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb25fe4c-923c-4e3c-8f83-fa9e6075e259 None admin] RBAC: Authorizing `identity:get_user(user_id=36edaa883d654921ac17e7745c0ff320)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:32.454132 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb25fe4c-923c-4e3c-8f83-fa9e6075e259 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:32.461032 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 40/81] 10.222.0.27 () {56 vars in 1177 bytes} [Sun Sep 11 00:38:32 2022] GET /identity/v3/users/36edaa883d654921ac17e7745c0ff320 => generated 247 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:32.461918 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb25fe4c-923c-4e3c-8f83-fa9e6075e259 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:32.491502 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-823a4e15-2e80-4b21-8833-bcc9649e7239 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:32.493034 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823a4e15-2e80-4b21-8833-bcc9649e7239 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:32.493034 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823a4e15-2e80-4b21-8833-bcc9649e7239 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:32.493034 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823a4e15-2e80-4b21-8833-bcc9649e7239 None admin] PATH_INFO: `/v3/users/2a97af87092c4282aa5985d5f454f26b` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:32.511913 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-823a4e15-2e80-4b21-8833-bcc9649e7239 None admin] RBAC: Authorizing `identity:get_user(user_id=2a97af87092c4282aa5985d5f454f26b)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:32.513450 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-823a4e15-2e80-4b21-8833-bcc9649e7239 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:32.515297 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 42/82] 10.222.0.27 () {56 vars in 1177 bytes} [Sun Sep 11 00:38:32 2022] GET /identity/v3/users/2a97af87092c4282aa5985d5f454f26b => generated 275 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:32.516060 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-823a4e15-2e80-4b21-8833-bcc9649e7239 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:32.592450 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb25fe4c-923c-4e3c-8f83-fa9e6075e259 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:32.638169 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-823a4e15-2e80-4b21-8833-bcc9649e7239 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:32.638463 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17bd9f49-1ddd-4a51-b56d-171aef38b231 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:32.639135 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17bd9f49-1ddd-4a51-b56d-171aef38b231 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:32.639358 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17bd9f49-1ddd-4a51-b56d-171aef38b231 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:32.639701 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17bd9f49-1ddd-4a51-b56d-171aef38b231 None admin] PATH_INFO: `/v3/users/2a97af87092c4282aa5985d5f454f26b` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:32.640787 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17bd9f49-1ddd-4a51-b56d-171aef38b231 None admin] RBAC: Authorizing `identity:get_user(user_id=2a97af87092c4282aa5985d5f454f26b)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:32.642262 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17bd9f49-1ddd-4a51-b56d-171aef38b231 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:32.643896 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 41/83] 10.222.0.27 () {56 vars in 1177 bytes} [Sun Sep 11 00:38:32 2022] GET /identity/v3/users/2a97af87092c4282aa5985d5f454f26b => generated 275 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:32.644667 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17bd9f49-1ddd-4a51-b56d-171aef38b231 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:32.668770 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:32.685097 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d499708-b803-4dca-9840-23b03050aa6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:32.685736 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d499708-b803-4dca-9840-23b03050aa6e None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:32.686003 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d499708-b803-4dca-9840-23b03050aa6e None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:32.686281 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d499708-b803-4dca-9840-23b03050aa6e None admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:32.686991 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d499708-b803-4dca-9840-23b03050aa6e None admin] RBAC: Adding query filter params (name=member) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:38:32.687642 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d499708-b803-4dca-9840-23b03050aa6e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:32.688699 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d499708-b803-4dca-9840-23b03050aa6e None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:32.701143 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 43/84] 10.222.0.27 () {56 vars in 1101 bytes} [Sun Sep 11 00:38:32 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:32.702473 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d499708-b803-4dca-9840-23b03050aa6e None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:32.743313 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:32.772666 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17bd9f49-1ddd-4a51-b56d-171aef38b231 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:32.817471 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba8fa2f2-6a1a-4c38-bef2-a8ad623c81c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:32.818584 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba8fa2f2-6a1a-4c38-bef2-a8ad623c81c4 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:32.822152 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba8fa2f2-6a1a-4c38-bef2-a8ad623c81c4 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:32.822962 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba8fa2f2-6a1a-4c38-bef2-a8ad623c81c4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:32.823495 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d499708-b803-4dca-9840-23b03050aa6e None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:32.861710 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c7178c-fdb2-4d6b-bb6c-56f9282075f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:32.862771 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c7178c-fdb2-4d6b-bb6c-56f9282075f0 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:32.863269 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c7178c-fdb2-4d6b-bb6c-56f9282075f0 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:32.863732 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c7178c-fdb2-4d6b-bb6c-56f9282075f0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:32.917121 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba8fa2f2-6a1a-4c38-bef2-a8ad623c81c4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:32.918692 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba8fa2f2-6a1a-4c38-bef2-a8ad623c81c4 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:32.940275 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3c7178c-fdb2-4d6b-bb6c-56f9282075f0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:32.940933 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:32.943258 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3c7178c-fdb2-4d6b-bb6c-56f9282075f0 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:32.967521 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:33.043732 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 42/85] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:38:32 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 399 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 11 00:38:33.044598 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba8fa2f2-6a1a-4c38-bef2-a8ad623c81c4 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:33.061119 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 44/86] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:38:32 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 359 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 11 00:38:33.062909 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c7178c-fdb2-4d6b-bb6c-56f9282075f0 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:33.163471 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba8fa2f2-6a1a-4c38-bef2-a8ad623c81c4 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:33.186561 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:33.194392 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c7178c-fdb2-4d6b-bb6c-56f9282075f0 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:33.205852 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2423db0c-c546-4d21-a637-3a41c8a36130 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:33.206467 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2423db0c-c546-4d21-a637-3a41c8a36130 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:33.206742 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2423db0c-c546-4d21-a637-3a41c8a36130 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:33.207039 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2423db0c-c546-4d21-a637-3a41c8a36130 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:33.217101 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:33.238149 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ca13616-7353-4665-a889-72a089c80826 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:33.238816 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca13616-7353-4665-a889-72a089c80826 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:33.239108 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca13616-7353-4665-a889-72a089c80826 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:33.239385 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca13616-7353-4665-a889-72a089c80826 None admin] PATH_INFO: `/v3/users/2a97af87092c4282aa5985d5f454f26b` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:33.241244 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ca13616-7353-4665-a889-72a089c80826 None admin] RBAC: Authorizing `identity:get_user(user_id=2a97af87092c4282aa5985d5f454f26b)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:33.243087 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ca13616-7353-4665-a889-72a089c80826 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:33.245485 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 45/87] 10.222.0.27 () {56 vars in 1177 bytes} [Sun Sep 11 00:38:33 2022] GET /identity/v3/users/2a97af87092c4282aa5985d5f454f26b => generated 275 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:33.246256 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ca13616-7353-4665-a889-72a089c80826 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:33.289205 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2423db0c-c546-4d21-a637-3a41c8a36130 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:33.291628 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2423db0c-c546-4d21-a637-3a41c8a36130 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:33.367702 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ca13616-7353-4665-a889-72a089c80826 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:33.398138 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 43/88] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:38:33 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 354 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 11 00:38:33.399339 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2423db0c-c546-4d21-a637-3a41c8a36130 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:33.408405 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f5edd4e-e5a3-45ba-8cb5-982fb2afb6b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:33.409089 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5edd4e-e5a3-45ba-8cb5-982fb2afb6b2 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:33.409370 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5edd4e-e5a3-45ba-8cb5-982fb2afb6b2 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:33.409641 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5edd4e-e5a3-45ba-8cb5-982fb2afb6b2 None admin] PATH_INFO: `/v3/projects/1cc0eb0de66448aa8d924179a50e25e9` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:33.411086 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f5edd4e-e5a3-45ba-8cb5-982fb2afb6b2 None admin] RBAC: Authorizing `identity:get_project(project_id=1cc0eb0de66448aa8d924179a50e25e9)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:33.412642 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f5edd4e-e5a3-45ba-8cb5-982fb2afb6b2 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:33.414612 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 46/89] 10.222.0.27 () {56 vars in 1186 bytes} [Sun Sep 11 00:38:33 2022] GET /identity/v3/projects/1cc0eb0de66448aa8d924179a50e25e9 => generated 298 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:33.415169 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f5edd4e-e5a3-45ba-8cb5-982fb2afb6b2 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:33.421488 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:33.481453 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:33.513942 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2423db0c-c546-4d21-a637-3a41c8a36130 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:33.529820 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f5edd4e-e5a3-45ba-8cb5-982fb2afb6b2 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:33.551769 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab1c3ea-0842-42f9-a4cc-61d3162df33c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:33.552630 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab1c3ea-0842-42f9-a4cc-61d3162df33c None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:33.553079 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab1c3ea-0842-42f9-a4cc-61d3162df33c None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:33.553518 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab1c3ea-0842-42f9-a4cc-61d3162df33c None admin] PATH_INFO: `/v3/projects/1cc0eb0de66448aa8d924179a50e25e9` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:33.554975 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab1c3ea-0842-42f9-a4cc-61d3162df33c None admin] RBAC: Authorizing `identity:get_project(project_id=1cc0eb0de66448aa8d924179a50e25e9)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:33.556843 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab1c3ea-0842-42f9-a4cc-61d3162df33c None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:33.558953 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 44/90] 10.222.0.27 () {56 vars in 1186 bytes} [Sun Sep 11 00:38:33 2022] GET /identity/v3/projects/1cc0eb0de66448aa8d924179a50e25e9 => generated 298 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:33.559920 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab1c3ea-0842-42f9-a4cc-61d3162df33c None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:33.567681 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0aa8b38-84dd-40e1-8a9d-27efdf0597f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:33.568301 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0aa8b38-84dd-40e1-8a9d-27efdf0597f7 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:33.568583 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0aa8b38-84dd-40e1-8a9d-27efdf0597f7 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:33.568859 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0aa8b38-84dd-40e1-8a9d-27efdf0597f7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:33.643651 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0aa8b38-84dd-40e1-8a9d-27efdf0597f7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:33.645276 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0aa8b38-84dd-40e1-8a9d-27efdf0597f7 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:33.652076 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:33.670427 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab1c3ea-0842-42f9-a4cc-61d3162df33c None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:33.707352 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe64515-e3f8-4305-950a-1eceaf362da1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:33.708519 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:33.708850 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe64515-e3f8-4305-950a-1eceaf362da1 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:33.709945 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe64515-e3f8-4305-950a-1eceaf362da1 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:33.710498 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe64515-e3f8-4305-950a-1eceaf362da1 None admin] PATH_INFO: `/v3/projects/1cc0eb0de66448aa8d924179a50e25e9` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:33.712036 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe64515-e3f8-4305-950a-1eceaf362da1 None admin] RBAC: Authorizing `identity:get_project(project_id=1cc0eb0de66448aa8d924179a50e25e9)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:33.715156 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe64515-e3f8-4305-950a-1eceaf362da1 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:33.717630 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 45/91] 10.222.0.27 () {56 vars in 1186 bytes} [Sun Sep 11 00:38:33 2022] GET /identity/v3/projects/1cc0eb0de66448aa8d924179a50e25e9 => generated 298 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:33.718591 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe64515-e3f8-4305-950a-1eceaf362da1 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:33.759135 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 47/92] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:38:33 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 344 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 11 00:38:33.760375 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0aa8b38-84dd-40e1-8a9d-27efdf0597f7 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:33.838621 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe64515-e3f8-4305-950a-1eceaf362da1 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:33.877613 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0aa8b38-84dd-40e1-8a9d-27efdf0597f7 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:33.884733 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473c33ac-41f9-48ba-aab4-0523713a2e5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:33.886376 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473c33ac-41f9-48ba-aab4-0523713a2e5b None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:33.886954 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473c33ac-41f9-48ba-aab4-0523713a2e5b None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:33.887430 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473c33ac-41f9-48ba-aab4-0523713a2e5b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:33.888801 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-473c33ac-41f9-48ba-aab4-0523713a2e5b None admin] RBAC: Adding query filter params (role.id=7d9213c1187c4b5182bf8132cf23ea19, scope.project.id=1cc0eb0de66448aa8d924179a50e25e9, user.id=36edaa883d654921ac17e7745c0ff320) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:38:33.889763 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-473c33ac-41f9-48ba-aab4-0523713a2e5b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:33.892003 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-473c33ac-41f9-48ba-aab4-0523713a2e5b None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:33.899715 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 46/93] 10.222.0.27 () {56 vars in 1374 bytes} [Sun Sep 11 00:38:33 2022] GET /identity/v3/role_assignments?user.id=36edaa883d654921ac17e7745c0ff320&scope.project.id=1cc0eb0de66448aa8d924179a50e25e9&role.id=7d9213c1187c4b5182bf8132cf23ea19 => generated 259 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:33.901357 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473c33ac-41f9-48ba-aab4-0523713a2e5b None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:33.926372 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01690c47-5d08-47e9-9c85-f575cc4fb6f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:33.930244 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01690c47-5d08-47e9-9c85-f575cc4fb6f2 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:33.930244 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01690c47-5d08-47e9-9c85-f575cc4fb6f2 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:33.930550 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01690c47-5d08-47e9-9c85-f575cc4fb6f2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:33.932277 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01690c47-5d08-47e9-9c85-f575cc4fb6f2 None admin] RBAC: Adding query filter params (role.id=2e64fc597b6447df99a7992d411d1a34, scope.project.id=1cc0eb0de66448aa8d924179a50e25e9, user.id=2a97af87092c4282aa5985d5f454f26b) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:38:33.932592 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01690c47-5d08-47e9-9c85-f575cc4fb6f2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:33.934084 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01690c47-5d08-47e9-9c85-f575cc4fb6f2 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:33.937676 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:33.956655 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 48/94] 10.222.0.27 () {56 vars in 1374 bytes} [Sun Sep 11 00:38:33 2022] GET /identity/v3/role_assignments?user.id=2a97af87092c4282aa5985d5f454f26b&scope.project.id=1cc0eb0de66448aa8d924179a50e25e9&role.id=2e64fc597b6447df99a7992d411d1a34 => generated 259 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:33.958376 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01690c47-5d08-47e9-9c85-f575cc4fb6f2 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:34.004473 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:34.140881 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473c33ac-41f9-48ba-aab4-0523713a2e5b None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:34.185425 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba6dd021-b3f2-4846-8472-e8a8b8cf24a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:34.186179 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba6dd021-b3f2-4846-8472-e8a8b8cf24a8 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:34.186395 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba6dd021-b3f2-4846-8472-e8a8b8cf24a8 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:34.186672 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba6dd021-b3f2-4846-8472-e8a8b8cf24a8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:34.187458 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba6dd021-b3f2-4846-8472-e8a8b8cf24a8 None admin] RBAC: Adding query filter params (role.id=518eab0f758241369edcdd997b98ad12, scope.project.id=1cc0eb0de66448aa8d924179a50e25e9, user.id=2a97af87092c4282aa5985d5f454f26b) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:38:34.187768 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba6dd021-b3f2-4846-8472-e8a8b8cf24a8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:34.189314 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba6dd021-b3f2-4846-8472-e8a8b8cf24a8 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:34.196118 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 47/95] 10.222.0.27 () {56 vars in 1374 bytes} [Sun Sep 11 00:38:33 2022] GET /identity/v3/role_assignments?user.id=2a97af87092c4282aa5985d5f454f26b&scope.project.id=1cc0eb0de66448aa8d924179a50e25e9&role.id=518eab0f758241369edcdd997b98ad12 => generated 259 bytes in 296 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:34.200983 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01690c47-5d08-47e9-9c85-f575cc4fb6f2 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:34.254145 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deb24202-9724-41d3-b3e2-149d2dae516d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:34.254839 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb24202-9724-41d3-b3e2-149d2dae516d None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:34.255128 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb24202-9724-41d3-b3e2-149d2dae516d None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:34.255402 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb24202-9724-41d3-b3e2-149d2dae516d None admin] PATH_INFO: `/v3/projects/6b1547a650214e0dbd243c050595fb86` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:34.256615 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deb24202-9724-41d3-b3e2-149d2dae516d None admin] RBAC: Authorizing `identity:get_project(project_id=6b1547a650214e0dbd243c050595fb86)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:34.258127 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deb24202-9724-41d3-b3e2-149d2dae516d None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:34.260519 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 49/96] 10.222.0.27 () {56 vars in 1186 bytes} [Sun Sep 11 00:38:33 2022] GET /identity/v3/projects/6b1547a650214e0dbd243c050595fb86 => generated 312 bytes in 303 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:34.266779 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba6dd021-b3f2-4846-8472-e8a8b8cf24a8 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:34.381796 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:34.409556 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba6dd021-b3f2-4846-8472-e8a8b8cf24a8 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:34.456498 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9151a36-0af6-4693-a197-4539a6a96498 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:34.457128 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9151a36-0af6-4693-a197-4539a6a96498 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:34.457408 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9151a36-0af6-4693-a197-4539a6a96498 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:34.457681 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9151a36-0af6-4693-a197-4539a6a96498 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:34.458420 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9151a36-0af6-4693-a197-4539a6a96498 None admin] RBAC: Adding query filter params (role.id=2e64fc597b6447df99a7992d411d1a34, scope.project.id=6b1547a650214e0dbd243c050595fb86, user.id=2a97af87092c4282aa5985d5f454f26b) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:38:34.458730 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9151a36-0af6-4693-a197-4539a6a96498 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:34.460344 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9151a36-0af6-4693-a197-4539a6a96498 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:34.470688 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 48/97] 10.222.0.27 () {56 vars in 1374 bytes} [Sun Sep 11 00:38:34 2022] GET /identity/v3/role_assignments?user.id=2a97af87092c4282aa5985d5f454f26b&scope.project.id=6b1547a650214e0dbd243c050595fb86&role.id=2e64fc597b6447df99a7992d411d1a34 => generated 259 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:34.889291 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa36380e-3b15-4886-8720-9b50a6334125 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:34.889658 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa36380e-3b15-4886-8720-9b50a6334125 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:34.889859 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa36380e-3b15-4886-8720-9b50a6334125 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:34.891292 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 50/98] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:38:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:38:34.897407 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6243b1-511e-432a-b8c3-f791c39d991d None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:34.897697 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6243b1-511e-432a-b8c3-f791c39d991d None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:34.897973 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6243b1-511e-432a-b8c3-f791c39d991d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:34.917943 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-cb6243b1-511e-432a-b8c3-f791c39d991d None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:38:34.919164 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-cb6243b1-511e-432a-b8c3-f791c39d991d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:38:35.024991 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 49/99] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:38:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 11 00:38:35.179986 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1271a1-f222-46fa-bffa-aa2cc77bfcce None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:35.180263 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1271a1-f222-46fa-bffa-aa2cc77bfcce None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:35.180462 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1271a1-f222-46fa-bffa-aa2cc77bfcce None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:35.182771 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 51/100] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:38:35 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:38:35.195822 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70262bcb-ce7c-45f1-b4f6-58fe3d81a78e None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:35.196735 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70262bcb-ce7c-45f1-b4f6-58fe3d81a78e None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:35.197316 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70262bcb-ce7c-45f1-b4f6-58fe3d81a78e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:35.223768 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-70262bcb-ce7c-45f1-b4f6-58fe3d81a78e None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:38:35.225282 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-70262bcb-ce7c-45f1-b4f6-58fe3d81a78e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:38:35.312363 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 50/101] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:38:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:38:35.545087 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb1271a1-f222-46fa-bffa-aa2cc77bfcce None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:35.667892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:35.688853 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb1271a1-f222-46fa-bffa-aa2cc77bfcce None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:35.733633 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfc42181-cebc-476f-8ca2-7488cad89229 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:35.734277 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc42181-cebc-476f-8ca2-7488cad89229 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:35.734556 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc42181-cebc-476f-8ca2-7488cad89229 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:35.734850 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc42181-cebc-476f-8ca2-7488cad89229 None admin] PATH_INFO: `/v3/domains/default` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:35.739464 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfc42181-cebc-476f-8ca2-7488cad89229 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:35.740933 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfc42181-cebc-476f-8ca2-7488cad89229 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:35.742852 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 52/102] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:38:35 2022] GET /identity/v3/domains/default => generated 199 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:35.749080 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70262bcb-ce7c-45f1-b4f6-58fe3d81a78e None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:35.753093 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:35.857271 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfc42181-cebc-476f-8ca2-7488cad89229 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:35.918010 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70262bcb-ce7c-45f1-b4f6-58fe3d81a78e None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:36.000594 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37c0cc75-87c2-4528-ad9a-f57ccbd2721d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:36.001221 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c0cc75-87c2-4528-ad9a-f57ccbd2721d None admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:36.001497 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c0cc75-87c2-4528-ad9a-f57ccbd2721d None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:36.001768 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c0cc75-87c2-4528-ad9a-f57ccbd2721d None admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:36.002420 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37c0cc75-87c2-4528-ad9a-f57ccbd2721d None admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:36.003921 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37c0cc75-87c2-4528-ad9a-f57ccbd2721d None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:36.053210 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 51/103] 10.222.0.27 () {60 vars in 1141 bytes} [Sun Sep 11 00:38:35 2022] POST /identity/v3/projects => generated 302 bytes in 305 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:38:36.058675 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfc42181-cebc-476f-8ca2-7488cad89229 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:36.111607 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f6d95da-7deb-4b16-b090-73add59e06cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:36.112252 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6d95da-7deb-4b16-b090-73add59e06cc None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:36.112533 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6d95da-7deb-4b16-b090-73add59e06cc None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:36.112865 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6d95da-7deb-4b16-b090-73add59e06cc None admin] PATH_INFO: `/v3/roles/reader` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:36.128500 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f6d95da-7deb-4b16-b090-73add59e06cc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:38:36.134449 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-2f6d95da-7deb-4b16-b090-73add59e06cc None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 11 00:38:36.134449 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:38:36.134449 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:38:36.134449 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:38:36.134449 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:38:36.134449 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:38:36.134449 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:38:36.134449 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:38:36.134449 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:38:36.134449 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:38:36.134449 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:38:36.134449 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:38:36.134449 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:38:36.134449 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:38:36.134449 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:38:36.134449 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:38:36.135948 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:38:36.135948 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:38:36.135948 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:38:36.135948 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:38:36.135948 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:38:36.135948 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:38:36.135948 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:38:36.135948 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:38:36.135948 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:38:36.135948 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:38:36.135948 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:38:36.135948 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:38:36.135948 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:38:36.135948 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:38:36.135948 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:38:36.135948 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:38:36.136708 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:38:36.136708 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:38:36.136708 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:38:36.136708 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:38:36.136708 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:38:36.136708 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:38:36.136708 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:38:36.136708 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:38:36.136708 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:38:36.136708 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:38:36.136708 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 11 00:38:36.136708 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:38:36.138011 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 53/104] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:38:35 2022] GET /identity/v3/roles/reader => generated 84 bytes in 280 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:38:36.142874 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37c0cc75-87c2-4528-ad9a-f57ccbd2721d None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:36.225095 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:36.275443 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37c0cc75-87c2-4528-ad9a-f57ccbd2721d None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:36.316135 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a3f3b34-5dd1-4301-9004-ac7a686a4f2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:36.317000 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3f3b34-5dd1-4301-9004-ac7a686a4f2f None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:36.317420 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3f3b34-5dd1-4301-9004-ac7a686a4f2f None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:36.317875 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3f3b34-5dd1-4301-9004-ac7a686a4f2f None admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:36.318695 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a3f3b34-5dd1-4301-9004-ac7a686a4f2f None admin] RBAC: Adding query filter params (name=reader) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:38:36.319161 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a3f3b34-5dd1-4301-9004-ac7a686a4f2f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:36.320768 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a3f3b34-5dd1-4301-9004-ac7a686a4f2f None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:36.331019 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 52/105] 10.222.0.27 () {56 vars in 1101 bytes} [Sun Sep 11 00:38:36 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:36.339875 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f6d95da-7deb-4b16-b090-73add59e06cc None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:36.358349 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:36.452546 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f6d95da-7deb-4b16-b090-73add59e06cc None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:36.496256 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba81714d-7c6f-4b1c-9198-fad1f18d7301 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:36.497226 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba81714d-7c6f-4b1c-9198-fad1f18d7301 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:36.497704 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba81714d-7c6f-4b1c-9198-fad1f18d7301 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:36.498143 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba81714d-7c6f-4b1c-9198-fad1f18d7301 None admin] PATH_INFO: `/v3/users/67cdfca3c1f54228aab73b276b6d34bb` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:36.516220 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba81714d-7c6f-4b1c-9198-fad1f18d7301 None admin] RBAC: Authorizing `identity:get_user(user_id=67cdfca3c1f54228aab73b276b6d34bb)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:36.517954 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba81714d-7c6f-4b1c-9198-fad1f18d7301 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:36.519844 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 54/106] 10.222.0.27 () {56 vars in 1177 bytes} [Sun Sep 11 00:38:36 2022] GET /identity/v3/users/67cdfca3c1f54228aab73b276b6d34bb => generated 289 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:36.527264 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a3f3b34-5dd1-4301-9004-ac7a686a4f2f None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:36.659153 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a3f3b34-5dd1-4301-9004-ac7a686a4f2f None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:36.682920 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:36.707962 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9cc6149-e0b4-4f30-bbdf-f557bed91624 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:36.708850 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9cc6149-e0b4-4f30-bbdf-f557bed91624 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:36.709324 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9cc6149-e0b4-4f30-bbdf-f557bed91624 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:36.709763 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9cc6149-e0b4-4f30-bbdf-f557bed91624 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:36.789784 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9cc6149-e0b4-4f30-bbdf-f557bed91624 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:36.792076 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9cc6149-e0b4-4f30-bbdf-f557bed91624 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:36.895538 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 53/107] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:38:36 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 11 00:38:36.903887 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba81714d-7c6f-4b1c-9198-fad1f18d7301 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:36.990215 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:37.031545 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba81714d-7c6f-4b1c-9198-fad1f18d7301 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:37.079571 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7da3f94-be81-4ae0-bc64-56216791ef14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:37.080226 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7da3f94-be81-4ae0-bc64-56216791ef14 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:37.080503 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7da3f94-be81-4ae0-bc64-56216791ef14 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:37.080772 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7da3f94-be81-4ae0-bc64-56216791ef14 None admin] PATH_INFO: `/v3/projects/1cc0eb0de66448aa8d924179a50e25e9` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:37.081910 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7da3f94-be81-4ae0-bc64-56216791ef14 None admin] RBAC: Authorizing `identity:get_project(project_id=1cc0eb0de66448aa8d924179a50e25e9)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:37.083388 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7da3f94-be81-4ae0-bc64-56216791ef14 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:37.085247 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 55/108] 10.222.0.27 () {56 vars in 1186 bytes} [Sun Sep 11 00:38:36 2022] GET /identity/v3/projects/1cc0eb0de66448aa8d924179a50e25e9 => generated 298 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:37.091195 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9cc6149-e0b4-4f30-bbdf-f557bed91624 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:37.115411 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:37.216891 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9cc6149-e0b4-4f30-bbdf-f557bed91624 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:37.262269 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec110cff-f2a0-41f6-93e7-4e41480f7deb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:37.262917 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec110cff-f2a0-41f6-93e7-4e41480f7deb None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:37.263081 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec110cff-f2a0-41f6-93e7-4e41480f7deb None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:37.263351 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec110cff-f2a0-41f6-93e7-4e41480f7deb None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:37.264096 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec110cff-f2a0-41f6-93e7-4e41480f7deb None admin] RBAC: Adding query filter params (role.id=7d473d5a7be94698b3b0ed9a2f499a56, scope.project.id=1cc0eb0de66448aa8d924179a50e25e9, user.id=67cdfca3c1f54228aab73b276b6d34bb) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:38:37.264403 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec110cff-f2a0-41f6-93e7-4e41480f7deb None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:37.265874 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec110cff-f2a0-41f6-93e7-4e41480f7deb None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:37.272816 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 54/109] 10.222.0.27 () {56 vars in 1374 bytes} [Sun Sep 11 00:38:37 2022] GET /identity/v3/role_assignments?user.id=67cdfca3c1f54228aab73b276b6d34bb&scope.project.id=1cc0eb0de66448aa8d924179a50e25e9&role.id=7d473d5a7be94698b3b0ed9a2f499a56 => generated 259 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:37.371871 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1732aa72-c9d1-4a13-9274-d4f9d7b779b6 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:37.372503 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1732aa72-c9d1-4a13-9274-d4f9d7b779b6 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:37.372984 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1732aa72-c9d1-4a13-9274-d4f9d7b779b6 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:37.374629 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 56/110] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:38:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 11 00:38:37.380741 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e9c720d-e79b-4823-949b-6c98f3caed38 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:37.381350 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e9c720d-e79b-4823-949b-6c98f3caed38 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:37.381824 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e9c720d-e79b-4823-949b-6c98f3caed38 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:37.411934 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-9e9c720d-e79b-4823-949b-6c98f3caed38 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:38:37.413470 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-9e9c720d-e79b-4823-949b-6c98f3caed38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:38:37.511781 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 55/111] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:38:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 11 00:38:37.576813 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c71eaa2-eee8-4848-95d0-2ee8b0235f33 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:37.577410 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c71eaa2-eee8-4848-95d0-2ee8b0235f33 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:37.577877 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c71eaa2-eee8-4848-95d0-2ee8b0235f33 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:37.579480 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 57/112] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:38:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:38:37.585473 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e567a4ec-87e1-429c-9106-d0f645acd5dd None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:37.586120 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e567a4ec-87e1-429c-9106-d0f645acd5dd None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:37.586606 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e567a4ec-87e1-429c-9106-d0f645acd5dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:37.609482 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-e567a4ec-87e1-429c-9106-d0f645acd5dd None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:38:37.611173 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-e567a4ec-87e1-429c-9106-d0f645acd5dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:38:37.621909 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:37.692738 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 56/113] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:38:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:38:37.696664 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b618f71-69c0-446c-bb1f-ffdef6fc2dd5 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:37.697875 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b618f71-69c0-446c-bb1f-ffdef6fc2dd5 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:37.698411 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b618f71-69c0-446c-bb1f-ffdef6fc2dd5 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:37.700011 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 58/114] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:38:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:38:37.705334 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f33117b-82ef-4e87-a28a-e600b9a781a8 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:37.705890 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f33117b-82ef-4e87-a28a-e600b9a781a8 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:37.706350 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f33117b-82ef-4e87-a28a-e600b9a781a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:37.725956 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-0f33117b-82ef-4e87-a28a-e600b9a781a8 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:38:37.727469 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-0f33117b-82ef-4e87-a28a-e600b9a781a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:38:37.732013 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2351bf22-62f5-4f04-a765-a6312954082a None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:37.732595 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2351bf22-62f5-4f04-a765-a6312954082a None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:37.733060 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2351bf22-62f5-4f04-a765-a6312954082a None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:37.734692 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 59/115] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:38:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 11 00:38:37.739888 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918ec505-0e81-4a53-bbf0-d84cfc0f9065 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:37.740398 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918ec505-0e81-4a53-bbf0-d84cfc0f9065 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:37.740896 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918ec505-0e81-4a53-bbf0-d84cfc0f9065 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:37.772305 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-918ec505-0e81-4a53-bbf0-d84cfc0f9065 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:38:37.774338 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-918ec505-0e81-4a53-bbf0-d84cfc0f9065 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:38:37.866170 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 57/116] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:38:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:38:37.896079 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 60/117] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:38:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 11 00:38:37.924680 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f33117b-82ef-4e87-a28a-e600b9a781a8 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:38.016873 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:38.058187 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f33117b-82ef-4e87-a28a-e600b9a781a8 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:38.093547 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0365c4c9-90fc-4ab6-8f4b-58b8f70e930e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:38.094188 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0365c4c9-90fc-4ab6-8f4b-58b8f70e930e None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:38.094469 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0365c4c9-90fc-4ab6-8f4b-58b8f70e930e None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:38.094743 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0365c4c9-90fc-4ab6-8f4b-58b8f70e930e None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:38.106320 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0365c4c9-90fc-4ab6-8f4b-58b8f70e930e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:38:38.111379 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-0365c4c9-90fc-4ab6-8f4b-58b8f70e930e None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 11 00:38:38.111379 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:38:38.111379 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:38:38.111379 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:38:38.111379 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:38:38.111379 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:38:38.111379 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:38:38.111379 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:38:38.111379 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:38:38.111379 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:38:38.111379 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:38:38.111379 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:38:38.111379 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:38:38.111379 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:38:38.111379 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:38:38.111379 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:38:38.112258 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:38:38.112258 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:38:38.112258 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:38:38.112258 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:38:38.112258 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:38:38.112258 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:38:38.112258 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:38:38.112258 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:38:38.112258 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:38:38.112258 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:38:38.112258 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:38:38.112258 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:38:38.112258 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:38:38.112258 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:38:38.112258 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:38:38.112258 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:38:38.113013 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:38:38.113013 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:38:38.113013 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:38:38.113013 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:38:38.113013 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:38:38.113013 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:38:38.113013 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:38:38.113013 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:38:38.113013 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:38:38.113013 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:38:38.113013 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 11 00:38:38.113013 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:38:38.113013 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 58/118] 10.222.0.27 () {56 vars in 1114 bytes} [Sun Sep 11 00:38:37 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:38:38.120566 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-918ec505-0e81-4a53-bbf0-d84cfc0f9065 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:38.203767 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0365c4c9-90fc-4ab6-8f4b-58b8f70e930e None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:38.204517 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:38.254522 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-918ec505-0e81-4a53-bbf0-d84cfc0f9065 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:38.324433 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b06c84d1-e45d-4661-9525-15ba4acd1ad6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:38.325177 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06c84d1-e45d-4661-9525-15ba4acd1ad6 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:38.325464 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06c84d1-e45d-4661-9525-15ba4acd1ad6 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:38.325847 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06c84d1-e45d-4661-9525-15ba4acd1ad6 None admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:38.326640 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b06c84d1-e45d-4661-9525-15ba4acd1ad6 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:38:38.328426 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0365c4c9-90fc-4ab6-8f4b-58b8f70e930e None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:38.330124 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b06c84d1-e45d-4661-9525-15ba4acd1ad6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:38.332478 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b06c84d1-e45d-4661-9525-15ba4acd1ad6 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:38.347728 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 61/119] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:38:38 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:38:38.350948 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b06c84d1-e45d-4661-9525-15ba4acd1ad6 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:38.371900 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:38.377721 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e36160af-26db-4acf-ae43-cef2637a8569 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:38.378373 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36160af-26db-4acf-ae43-cef2637a8569 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:38.378652 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36160af-26db-4acf-ae43-cef2637a8569 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:38.378960 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36160af-26db-4acf-ae43-cef2637a8569 None admin] PATH_INFO: `/v3/roles/member` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:38.390963 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e36160af-26db-4acf-ae43-cef2637a8569 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:38:38.398109 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-e36160af-26db-4acf-ae43-cef2637a8569 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 11 00:38:38.398109 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:38:38.398109 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:38:38.398109 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:38:38.398109 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:38:38.398109 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:38:38.398109 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:38:38.398109 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:38:38.398109 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:38:38.398109 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:38:38.398109 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:38:38.398109 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:38:38.398109 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:38:38.398109 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:38:38.398109 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:38:38.398109 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:38:38.399226 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:38:38.399226 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:38:38.399226 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:38:38.399226 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:38:38.399226 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:38:38.399226 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:38:38.399226 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:38:38.399226 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:38:38.399226 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:38:38.399226 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:38:38.399226 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:38:38.399226 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:38:38.399226 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:38:38.399226 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:38:38.399226 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:38:38.399226 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:38:38.400122 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:38:38.400122 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:38:38.400122 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:38:38.400122 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:38:38.400122 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:38:38.400122 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:38:38.400122 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:38:38.400122 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:38:38.400122 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:38:38.400122 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:38:38.400122 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 11 00:38:38.400122 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:38:38.400122 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 59/120] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:38:38 2022] GET /identity/v3/roles/member => generated 84 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:38:38.400122 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e36160af-26db-4acf-ae43-cef2637a8569 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:38.474966 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b06c84d1-e45d-4661-9525-15ba4acd1ad6 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:38.507755 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:38.515284 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a78e4b-e0c9-417f-9fb4-5e715e6b5ccf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:38.516207 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a78e4b-e0c9-417f-9fb4-5e715e6b5ccf None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:38.516688 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a78e4b-e0c9-417f-9fb4-5e715e6b5ccf None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:38.517157 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a78e4b-e0c9-417f-9fb4-5e715e6b5ccf None admin] PATH_INFO: `/v3/roles/admin` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:38.528765 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24a78e4b-e0c9-417f-9fb4-5e715e6b5ccf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:38:38.534567 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-24a78e4b-e0c9-417f-9fb4-5e715e6b5ccf None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:38:38.534567 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:38:38.534567 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:38:38.534567 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:38:38.534567 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:38:38.534567 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:38:38.534567 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:38:38.534567 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:38:38.534567 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:38:38.534567 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:38:38.534567 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:38:38.534567 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:38:38.534567 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:38:38.534567 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:38:38.534567 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:38:38.534567 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:38:38.535703 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:38:38.535703 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:38:38.535703 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:38:38.535703 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:38:38.535703 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:38:38.535703 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:38:38.535703 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:38:38.535703 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:38:38.535703 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:38:38.535703 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:38:38.535703 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:38:38.535703 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:38:38.535703 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:38:38.535703 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:38:38.535703 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:38:38.535703 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:38:38.536468 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:38:38.536468 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:38:38.536468 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:38:38.536468 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:38:38.536468 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:38:38.536468 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:38:38.536468 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:38:38.536468 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:38:38.536468 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:38:38.536468 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:38:38.536468 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:38:38.536468 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:38:38.538752 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 62/121] 10.222.0.27 () {56 vars in 1096 bytes} [Sun Sep 11 00:38:38 2022] GET /identity/v3/roles/admin => generated 83 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:38:38.539682 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e36160af-26db-4acf-ae43-cef2637a8569 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:38.539969 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a78e4b-e0c9-417f-9fb4-5e715e6b5ccf None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:38.588368 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfda2cd9-a0b9-4bda-9fb8-1f37332a838b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:38.589023 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfda2cd9-a0b9-4bda-9fb8-1f37332a838b None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:38.589308 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfda2cd9-a0b9-4bda-9fb8-1f37332a838b None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:38.589587 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfda2cd9-a0b9-4bda-9fb8-1f37332a838b None admin] PATH_INFO: `/v3/roles/member` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:38.605505 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfda2cd9-a0b9-4bda-9fb8-1f37332a838b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:38:38.613041 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-bfda2cd9-a0b9-4bda-9fb8-1f37332a838b None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 11 00:38:38.613041 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:38:38.613041 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:38:38.613041 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:38:38.613041 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:38:38.613041 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:38:38.613041 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:38:38.613041 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:38:38.613041 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:38:38.613041 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:38:38.613041 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:38:38.613041 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:38:38.613041 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:38:38.613041 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:38:38.613041 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:38:38.613041 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:38:38.613992 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:38:38.613992 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:38:38.613992 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:38:38.613992 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:38:38.613992 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:38:38.613992 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:38:38.613992 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:38:38.613992 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:38:38.613992 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:38:38.613992 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:38:38.613992 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:38:38.613992 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:38:38.613992 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:38:38.613992 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:38:38.613992 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:38:38.613992 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:38:38.614829 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:38:38.614829 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:38:38.614829 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:38:38.614829 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:38:38.614829 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:38:38.614829 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:38:38.614829 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:38:38.614829 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:38:38.614829 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:38:38.614829 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:38:38.614829 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 11 00:38:38.614829 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:38:38.614829 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 60/122] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:38:38 2022] GET /identity/v3/roles/member => generated 84 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:38:38.615485 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfda2cd9-a0b9-4bda-9fb8-1f37332a838b None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:38.673788 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a78e4b-e0c9-417f-9fb4-5e715e6b5ccf None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:38.677558 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:38.718589 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f39e4403-afc0-47bc-a749-d83989f822b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:38.719572 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f39e4403-afc0-47bc-a749-d83989f822b8 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:38.719913 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f39e4403-afc0-47bc-a749-d83989f822b8 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:38.720198 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f39e4403-afc0-47bc-a749-d83989f822b8 None admin] PATH_INFO: `/v3/users/2a97af87092c4282aa5985d5f454f26b` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:38.721627 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f39e4403-afc0-47bc-a749-d83989f822b8 None admin] RBAC: Authorizing `identity:get_user(user_id=2a97af87092c4282aa5985d5f454f26b)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:38.723716 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f39e4403-afc0-47bc-a749-d83989f822b8 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:38.725742 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 63/123] 10.222.0.27 () {56 vars in 1177 bytes} [Sun Sep 11 00:38:38 2022] GET /identity/v3/users/2a97af87092c4282aa5985d5f454f26b => generated 275 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:38.730940 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f39e4403-afc0-47bc-a749-d83989f822b8 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:38.734287 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfda2cd9-a0b9-4bda-9fb8-1f37332a838b None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:38.769770 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e76f227-58e4-40d4-aad0-4b98ac601210 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:38.770709 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e76f227-58e4-40d4-aad0-4b98ac601210 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:38.771214 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e76f227-58e4-40d4-aad0-4b98ac601210 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:38.771684 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e76f227-58e4-40d4-aad0-4b98ac601210 None admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:38.772545 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e76f227-58e4-40d4-aad0-4b98ac601210 None admin] RBAC: Adding query filter params (name=member) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:38:38.773043 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e76f227-58e4-40d4-aad0-4b98ac601210 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:38.774669 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e76f227-58e4-40d4-aad0-4b98ac601210 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:38.783884 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 61/124] 10.222.0.27 () {56 vars in 1101 bytes} [Sun Sep 11 00:38:38 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:38.784974 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e76f227-58e4-40d4-aad0-4b98ac601210 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:38.801030 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:38.855444 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f39e4403-afc0-47bc-a749-d83989f822b8 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:38.895235 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e76f227-58e4-40d4-aad0-4b98ac601210 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:38.904870 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec1b964-8b19-43a8-b73c-7f6b312aefa0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:38.905555 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec1b964-8b19-43a8-b73c-7f6b312aefa0 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:38.905846 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec1b964-8b19-43a8-b73c-7f6b312aefa0 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:38.906123 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec1b964-8b19-43a8-b73c-7f6b312aefa0 None admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:38.907348 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec1b964-8b19-43a8-b73c-7f6b312aefa0 None admin] RBAC: Adding query filter params (name=admin) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:38:38.907708 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec1b964-8b19-43a8-b73c-7f6b312aefa0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:38.909696 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec1b964-8b19-43a8-b73c-7f6b312aefa0 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:38.912966 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:38.921048 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 64/125] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:38:38 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:38.930910 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec1b964-8b19-43a8-b73c-7f6b312aefa0 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:38.939689 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a31f0e-35da-44b8-95c9-f09b2a9d4991 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:38.940621 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a31f0e-35da-44b8-95c9-f09b2a9d4991 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:38.941113 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a31f0e-35da-44b8-95c9-f09b2a9d4991 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:38.941572 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a31f0e-35da-44b8-95c9-f09b2a9d4991 None admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:38.942480 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a31f0e-35da-44b8-95c9-f09b2a9d4991 None admin] RBAC: Adding query filter params (name=member) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:38:38.943037 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a31f0e-35da-44b8-95c9-f09b2a9d4991 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:38.944610 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a31f0e-35da-44b8-95c9-f09b2a9d4991 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:38.953945 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 62/126] 10.222.0.27 () {56 vars in 1101 bytes} [Sun Sep 11 00:38:38 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:38.955129 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a31f0e-35da-44b8-95c9-f09b2a9d4991 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:39.055279 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec1b964-8b19-43a8-b73c-7f6b312aefa0 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:39.066060 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a31f0e-35da-44b8-95c9-f09b2a9d4991 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:39.085546 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:39.103947 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d290a48b-0fbb-4fe3-8ace-7db8074bb850 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:39.104613 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d290a48b-0fbb-4fe3-8ace-7db8074bb850 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:39.104901 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d290a48b-0fbb-4fe3-8ace-7db8074bb850 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:39.105177 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d290a48b-0fbb-4fe3-8ace-7db8074bb850 None admin] PATH_INFO: `/v3/users/2a97af87092c4282aa5985d5f454f26b` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:39.106323 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a008741-d4d6-4bf7-b319-f5d238a18294 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:39.107006 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a008741-d4d6-4bf7-b319-f5d238a18294 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:39.107284 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a008741-d4d6-4bf7-b319-f5d238a18294 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:39.107559 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a008741-d4d6-4bf7-b319-f5d238a18294 None admin] PATH_INFO: `/v3/projects/1cc0eb0de66448aa8d924179a50e25e9` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:39.108128 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d290a48b-0fbb-4fe3-8ace-7db8074bb850 None admin] RBAC: Authorizing `identity:get_user(user_id=2a97af87092c4282aa5985d5f454f26b)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:39.108716 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a008741-d4d6-4bf7-b319-f5d238a18294 None admin] RBAC: Authorizing `identity:get_project(project_id=1cc0eb0de66448aa8d924179a50e25e9)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:39.110071 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d290a48b-0fbb-4fe3-8ace-7db8074bb850 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:39.110175 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a008741-d4d6-4bf7-b319-f5d238a18294 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:39.111970 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 65/127] 10.222.0.27 () {56 vars in 1186 bytes} [Sun Sep 11 00:38:38 2022] GET /identity/v3/projects/1cc0eb0de66448aa8d924179a50e25e9 => generated 298 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:39.112565 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 63/128] 10.222.0.27 () {56 vars in 1177 bytes} [Sun Sep 11 00:38:38 2022] GET /identity/v3/users/2a97af87092c4282aa5985d5f454f26b => generated 275 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:39.112666 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a008741-d4d6-4bf7-b319-f5d238a18294 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:39.118101 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d290a48b-0fbb-4fe3-8ace-7db8074bb850 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:39.168131 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:39.233052 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d290a48b-0fbb-4fe3-8ace-7db8074bb850 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:39.241407 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a008741-d4d6-4bf7-b319-f5d238a18294 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:39.275490 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f60a1875-7a40-4d01-ad50-63de41cf06b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:39.277036 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60a1875-7a40-4d01-ad50-63de41cf06b3 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:39.277463 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60a1875-7a40-4d01-ad50-63de41cf06b3 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:39.277874 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60a1875-7a40-4d01-ad50-63de41cf06b3 None admin] PATH_INFO: `/v3/users/2a97af87092c4282aa5985d5f454f26b` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:39.279663 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f60a1875-7a40-4d01-ad50-63de41cf06b3 None admin] RBAC: Authorizing `identity:get_user(user_id=2a97af87092c4282aa5985d5f454f26b)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:39.281531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f60a1875-7a40-4d01-ad50-63de41cf06b3 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:39.283698 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 64/129] 10.222.0.27 () {56 vars in 1177 bytes} [Sun Sep 11 00:38:39 2022] GET /identity/v3/users/2a97af87092c4282aa5985d5f454f26b => generated 275 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:39.284608 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a1cb5b-d10c-4087-8584-f1600f6181e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:39.284843 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f60a1875-7a40-4d01-ad50-63de41cf06b3 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:39.285836 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a1cb5b-d10c-4087-8584-f1600f6181e5 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:39.286226 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a1cb5b-d10c-4087-8584-f1600f6181e5 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:39.286610 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a1cb5b-d10c-4087-8584-f1600f6181e5 None admin] PATH_INFO: `/v3/users/36edaa883d654921ac17e7745c0ff320` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:39.287906 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55a1cb5b-d10c-4087-8584-f1600f6181e5 None admin] RBAC: Authorizing `identity:get_user(user_id=36edaa883d654921ac17e7745c0ff320)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:39.291347 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55a1cb5b-d10c-4087-8584-f1600f6181e5 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:39.293256 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 66/130] 10.222.0.27 () {56 vars in 1177 bytes} [Sun Sep 11 00:38:39 2022] GET /identity/v3/users/36edaa883d654921ac17e7745c0ff320 => generated 247 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:39.299026 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a1cb5b-d10c-4087-8584-f1600f6181e5 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:39.373449 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:39.415350 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f60a1875-7a40-4d01-ad50-63de41cf06b3 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:39.431452 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a1cb5b-d10c-4087-8584-f1600f6181e5 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:39.433330 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:39.466934 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cb3ee27-7b22-4af9-9fe6-174a3f485a11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:39.468174 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb3ee27-7b22-4af9-9fe6-174a3f485a11 None admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:39.468603 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb3ee27-7b22-4af9-9fe6-174a3f485a11 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:39.469021 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb3ee27-7b22-4af9-9fe6-174a3f485a11 None admin] PATH_INFO: `/v3/projects/1cc0eb0de66448aa8d924179a50e25e9/users/2a97af87092c4282aa5985d5f454f26b/roles/518eab0f758241369edcdd997b98ad12` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:39.472592 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cb3ee27-7b22-4af9-9fe6-174a3f485a11 None admin] RBAC: Authorizing `identity:create_grant(project_id=1cc0eb0de66448aa8d924179a50e25e9, user_id=2a97af87092c4282aa5985d5f454f26b, role_id=518eab0f758241369edcdd997b98ad12)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:39.474919 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cb3ee27-7b22-4af9-9fe6-174a3f485a11 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:39.485636 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d82f776-362a-4fff-92b2-85a3107d1fdd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:39.486447 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d82f776-362a-4fff-92b2-85a3107d1fdd None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:39.486686 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d82f776-362a-4fff-92b2-85a3107d1fdd None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:39.487131 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d82f776-362a-4fff-92b2-85a3107d1fdd None admin] PATH_INFO: `/v3/projects/1cc0eb0de66448aa8d924179a50e25e9` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:39.488367 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d82f776-362a-4fff-92b2-85a3107d1fdd None admin] RBAC: Authorizing `identity:get_project(project_id=1cc0eb0de66448aa8d924179a50e25e9)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:39.490756 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d82f776-362a-4fff-92b2-85a3107d1fdd None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:39.492878 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 67/131] 10.222.0.27 () {56 vars in 1186 bytes} [Sun Sep 11 00:38:39 2022] GET /identity/v3/projects/1cc0eb0de66448aa8d924179a50e25e9 => generated 298 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:39.493814 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d82f776-362a-4fff-92b2-85a3107d1fdd None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:39.495670 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 65/132] 10.222.0.27 () {58 vars in 1439 bytes} [Sun Sep 11 00:38:39 2022] PUT /identity/v3/projects/1cc0eb0de66448aa8d924179a50e25e9/users/2a97af87092c4282aa5985d5f454f26b/roles/518eab0f758241369edcdd997b98ad12 => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:38:39.496869 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cb3ee27-7b22-4af9-9fe6-174a3f485a11 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:39.623497 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d82f776-362a-4fff-92b2-85a3107d1fdd None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:39.639378 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cb3ee27-7b22-4af9-9fe6-174a3f485a11 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:39.655774 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:39.668261 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6e494a9-9bed-4979-b172-7d7d08b29699 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:39.669097 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e494a9-9bed-4979-b172-7d7d08b29699 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:39.669483 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e494a9-9bed-4979-b172-7d7d08b29699 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:39.669864 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e494a9-9bed-4979-b172-7d7d08b29699 None admin] PATH_INFO: `/v3/projects/6b1547a650214e0dbd243c050595fb86` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:39.671283 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6e494a9-9bed-4979-b172-7d7d08b29699 None admin] RBAC: Authorizing `identity:get_project(project_id=6b1547a650214e0dbd243c050595fb86)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:39.673862 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6e494a9-9bed-4979-b172-7d7d08b29699 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:39.675822 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 68/133] 10.222.0.27 () {56 vars in 1186 bytes} [Sun Sep 11 00:38:39 2022] GET /identity/v3/projects/6b1547a650214e0dbd243c050595fb86 => generated 312 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:39.676773 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6e494a9-9bed-4979-b172-7d7d08b29699 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:39.694092 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-821a7ea4-7289-434e-8b75-92af4f3a3918 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:39.694998 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821a7ea4-7289-434e-8b75-92af4f3a3918 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:39.695309 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821a7ea4-7289-434e-8b75-92af4f3a3918 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:39.695594 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821a7ea4-7289-434e-8b75-92af4f3a3918 None admin] PATH_INFO: `/v3/projects/1cc0eb0de66448aa8d924179a50e25e9` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:39.697807 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-821a7ea4-7289-434e-8b75-92af4f3a3918 None admin] RBAC: Authorizing `identity:get_project(project_id=1cc0eb0de66448aa8d924179a50e25e9)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:39.699420 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-821a7ea4-7289-434e-8b75-92af4f3a3918 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:39.701349 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 66/134] 10.222.0.27 () {56 vars in 1186 bytes} [Sun Sep 11 00:38:39 2022] GET /identity/v3/projects/1cc0eb0de66448aa8d924179a50e25e9 => generated 298 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:39.707563 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4271a18-63b3-4a74-ba9f-47164c386a2b None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:39.707856 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4271a18-63b3-4a74-ba9f-47164c386a2b None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:39.708134 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4271a18-63b3-4a74-ba9f-47164c386a2b None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:39.709503 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 67/135] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:38:39 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:38:39.710782 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4271a18-63b3-4a74-ba9f-47164c386a2b None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:39.772938 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:39.806372 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6e494a9-9bed-4979-b172-7d7d08b29699 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:39.846710 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4271a18-63b3-4a74-ba9f-47164c386a2b None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:39.858162 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a5b578-accb-4091-b933-4031a6b22ce8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:39.859728 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a5b578-accb-4091-b933-4031a6b22ce8 None admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:39.860195 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a5b578-accb-4091-b933-4031a6b22ce8 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:39.860629 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a5b578-accb-4091-b933-4031a6b22ce8 None admin] PATH_INFO: `/v3/projects/1cc0eb0de66448aa8d924179a50e25e9/users/2a97af87092c4282aa5985d5f454f26b/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:39.863804 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43a5b578-accb-4091-b933-4031a6b22ce8 None admin] RBAC: Authorizing `identity:create_grant(project_id=1cc0eb0de66448aa8d924179a50e25e9, user_id=2a97af87092c4282aa5985d5f454f26b, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:39.865723 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43a5b578-accb-4091-b933-4031a6b22ce8 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:39.891033 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 69/136] 10.222.0.27 () {58 vars in 1439 bytes} [Sun Sep 11 00:38:39 2022] PUT /identity/v3/projects/1cc0eb0de66448aa8d924179a50e25e9/users/2a97af87092c4282aa5985d5f454f26b/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:38:39.893851 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-587f30b6-ed37-4462-a392-b46f6e8a54fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:39.894886 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587f30b6-ed37-4462-a392-b46f6e8a54fd None admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:39.895336 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587f30b6-ed37-4462-a392-b46f6e8a54fd None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:39.895832 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587f30b6-ed37-4462-a392-b46f6e8a54fd None admin] PATH_INFO: `/v3/projects/6b1547a650214e0dbd243c050595fb86/users/2a97af87092c4282aa5985d5f454f26b/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:39.898237 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-587f30b6-ed37-4462-a392-b46f6e8a54fd None admin] RBAC: Authorizing `identity:create_grant(project_id=6b1547a650214e0dbd243c050595fb86, user_id=2a97af87092c4282aa5985d5f454f26b, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:39.899100 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a5b578-accb-4091-b933-4031a6b22ce8 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:39.900489 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-587f30b6-ed37-4462-a392-b46f6e8a54fd None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:39.930056 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 68/137] 10.222.0.27 () {58 vars in 1439 bytes} [Sun Sep 11 00:38:39 2022] PUT /identity/v3/projects/6b1547a650214e0dbd243c050595fb86/users/2a97af87092c4282aa5985d5f454f26b/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 220 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:38:39.936177 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce346924-29f5-4212-8570-89718d1ed098 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:39.939309 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce346924-29f5-4212-8570-89718d1ed098 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:39.939844 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce346924-29f5-4212-8570-89718d1ed098 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:39.989476 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-ce346924-29f5-4212-8570-89718d1ed098 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:38:39.992492 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-ce346924-29f5-4212-8570-89718d1ed098 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:38:40.037331 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:40.101537 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 69/138] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:38:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:38:40.103717 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20f9843-51fb-4a26-863e-9f1deba6f238 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:40.104207 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20f9843-51fb-4a26-863e-9f1deba6f238 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:40.105255 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20f9843-51fb-4a26-863e-9f1deba6f238 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:40.107585 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 70/139] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:38:40 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:38:40.116458 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6aa6cf-effb-49e0-8815-5f899f68c1df None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:40.117107 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6aa6cf-effb-49e0-8815-5f899f68c1df None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:40.117862 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6aa6cf-effb-49e0-8815-5f899f68c1df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:40.149104 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a5b578-accb-4091-b933-4031a6b22ce8 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:40.152437 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-1c6aa6cf-effb-49e0-8815-5f899f68c1df None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:38:40.153858 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-1c6aa6cf-effb-49e0-8815-5f899f68c1df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:38:40.190408 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a30933f2-9dba-4e66-8c5f-e998508c0b39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:40.191142 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a30933f2-9dba-4e66-8c5f-e998508c0b39 None admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:40.191483 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a30933f2-9dba-4e66-8c5f-e998508c0b39 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:40.192675 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a30933f2-9dba-4e66-8c5f-e998508c0b39 None admin] PATH_INFO: `/v3/projects/1cc0eb0de66448aa8d924179a50e25e9/users/36edaa883d654921ac17e7745c0ff320/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:40.193961 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a30933f2-9dba-4e66-8c5f-e998508c0b39 None admin] RBAC: Authorizing `identity:create_grant(project_id=1cc0eb0de66448aa8d924179a50e25e9, user_id=36edaa883d654921ac17e7745c0ff320, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:40.195581 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a30933f2-9dba-4e66-8c5f-e998508c0b39 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:40.197978 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:40.219832 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 70/140] 10.222.0.27 () {58 vars in 1439 bytes} [Sun Sep 11 00:38:39 2022] PUT /identity/v3/projects/1cc0eb0de66448aa8d924179a50e25e9/users/36edaa883d654921ac17e7745c0ff320/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 326 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:38:40.268933 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 71/141] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:38:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:38:40.412211 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a30933f2-9dba-4e66-8c5f-e998508c0b39 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:40.533754 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a30933f2-9dba-4e66-8c5f-e998508c0b39 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:40.551872 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:40.574742 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-712cd4b2-3b6f-46a0-b787-4c224db4557c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:40.575412 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712cd4b2-3b6f-46a0-b787-4c224db4557c None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:40.575696 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712cd4b2-3b6f-46a0-b787-4c224db4557c None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:40.575972 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712cd4b2-3b6f-46a0-b787-4c224db4557c None admin] PATH_INFO: `/v3/domains/default` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:40.577107 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-712cd4b2-3b6f-46a0-b787-4c224db4557c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:40.578564 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-712cd4b2-3b6f-46a0-b787-4c224db4557c None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:40.580393 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 71/142] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:38:40 2022] GET /identity/v3/domains/default => generated 199 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:40.585723 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c6aa6cf-effb-49e0-8815-5f899f68c1df None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:40.726930 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c6aa6cf-effb-49e0-8815-5f899f68c1df None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:40.768693 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-712cd4b2-3b6f-46a0-b787-4c224db4557c None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:40.779164 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99cba5c9-190b-431e-a576-a695beb91e03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:40.780092 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99cba5c9-190b-431e-a576-a695beb91e03 None admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:40.780568 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99cba5c9-190b-431e-a576-a695beb91e03 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:40.781054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99cba5c9-190b-431e-a576-a695beb91e03 None admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:40.781942 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99cba5c9-190b-431e-a576-a695beb91e03 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:40.783620 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99cba5c9-190b-431e-a576-a695beb91e03 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:40.816735 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 72/143] 10.222.0.27 () {60 vars in 1132 bytes} [Sun Sep 11 00:38:40 2022] POST /identity/v3/users => generated 283 bytes in 232 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:38:40.936011 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-712cd4b2-3b6f-46a0-b787-4c224db4557c None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:40.979006 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca4e5b9-46e6-4b98-a6fc-a407a1e6e670 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:40.979583 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca4e5b9-46e6-4b98-a6fc-a407a1e6e670 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:40.979865 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca4e5b9-46e6-4b98-a6fc-a407a1e6e670 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:40.980137 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca4e5b9-46e6-4b98-a6fc-a407a1e6e670 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:40.997273 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ca4e5b9-46e6-4b98-a6fc-a407a1e6e670 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:38:41.002728 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-2ca4e5b9-46e6-4b98-a6fc-a407a1e6e670 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 11 00:38:41.002728 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:38:41.002728 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:38:41.002728 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:38:41.002728 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:38:41.002728 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:38:41.002728 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:38:41.002728 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:38:41.002728 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:38:41.002728 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:38:41.002728 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:38:41.002728 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:38:41.002728 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:38:41.002728 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:38:41.002728 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:38:41.002728 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:38:41.003593 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:38:41.003593 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:38:41.003593 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:38:41.003593 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:38:41.003593 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:38:41.003593 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:38:41.003593 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:38:41.003593 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:38:41.003593 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:38:41.003593 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:38:41.003593 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:38:41.003593 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:38:41.003593 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:38:41.003593 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:38:41.003593 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:38:41.003593 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:38:41.004775 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:38:41.004775 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:38:41.004775 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:38:41.004775 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:38:41.004775 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:38:41.004775 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:38:41.004775 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:38:41.004775 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:38:41.004775 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:38:41.004775 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:38:41.004775 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 11 00:38:41.004775 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:38:41.004775 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 72/144] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:38:40 2022] GET /identity/v3/roles/reader => generated 84 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:38:41.009928 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99cba5c9-190b-431e-a576-a695beb91e03 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:41.040612 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:41.167366 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99cba5c9-190b-431e-a576-a695beb91e03 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:41.204453 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6c97da5-f65c-4148-ae52-3ed3643bd5ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:41.205065 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c97da5-f65c-4148-ae52-3ed3643bd5ea None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:41.205337 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c97da5-f65c-4148-ae52-3ed3643bd5ea None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:41.205602 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c97da5-f65c-4148-ae52-3ed3643bd5ea None admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:41.206278 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6c97da5-f65c-4148-ae52-3ed3643bd5ea None admin] RBAC: Adding query filter params (name=reader) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:38:41.206576 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6c97da5-f65c-4148-ae52-3ed3643bd5ea None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:41.207977 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6c97da5-f65c-4148-ae52-3ed3643bd5ea None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:41.217051 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 73/145] 10.222.0.27 () {56 vars in 1101 bytes} [Sun Sep 11 00:38:41 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:41.222562 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca4e5b9-46e6-4b98-a6fc-a407a1e6e670 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:41.285320 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:41.357175 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca4e5b9-46e6-4b98-a6fc-a407a1e6e670 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:41.400763 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78787917-eff6-4064-85b5-81cb8d536e08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:41.401411 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78787917-eff6-4064-85b5-81cb8d536e08 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:41.401697 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78787917-eff6-4064-85b5-81cb8d536e08 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:41.401971 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78787917-eff6-4064-85b5-81cb8d536e08 None admin] PATH_INFO: `/v3/users/67cdfca3c1f54228aab73b276b6d34bb` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:41.403219 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78787917-eff6-4064-85b5-81cb8d536e08 None admin] RBAC: Authorizing `identity:get_user(user_id=67cdfca3c1f54228aab73b276b6d34bb)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:41.404693 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78787917-eff6-4064-85b5-81cb8d536e08 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:41.406273 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 73/146] 10.222.0.27 () {56 vars in 1177 bytes} [Sun Sep 11 00:38:41 2022] GET /identity/v3/users/67cdfca3c1f54228aab73b276b6d34bb => generated 289 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:41.412101 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6c97da5-f65c-4148-ae52-3ed3643bd5ea None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:41.541097 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:41.574296 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6c97da5-f65c-4148-ae52-3ed3643bd5ea None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:41.627120 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1920bc15-7df9-412c-8c0a-f905c952555d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:41.627765 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1920bc15-7df9-412c-8c0a-f905c952555d None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:41.628018 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1920bc15-7df9-412c-8c0a-f905c952555d None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:41.628288 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1920bc15-7df9-412c-8c0a-f905c952555d None admin] PATH_INFO: `/v3/projects/1cc0eb0de66448aa8d924179a50e25e9` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:41.629446 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1920bc15-7df9-412c-8c0a-f905c952555d None admin] RBAC: Authorizing `identity:get_project(project_id=1cc0eb0de66448aa8d924179a50e25e9)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:41.630950 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1920bc15-7df9-412c-8c0a-f905c952555d None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:41.632598 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 74/147] 10.222.0.27 () {56 vars in 1186 bytes} [Sun Sep 11 00:38:41 2022] GET /identity/v3/projects/1cc0eb0de66448aa8d924179a50e25e9 => generated 298 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:41.639014 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78787917-eff6-4064-85b5-81cb8d536e08 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:41.779737 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78787917-eff6-4064-85b5-81cb8d536e08 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:41.793697 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:41.828694 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64477d2a-6239-4ecb-821d-c093c73716aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:41.829409 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64477d2a-6239-4ecb-821d-c093c73716aa None admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:41.829739 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64477d2a-6239-4ecb-821d-c093c73716aa None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:41.830017 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64477d2a-6239-4ecb-821d-c093c73716aa None admin] PATH_INFO: `/v3/projects/1cc0eb0de66448aa8d924179a50e25e9/users/67cdfca3c1f54228aab73b276b6d34bb/roles/7d473d5a7be94698b3b0ed9a2f499a56` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:41.832007 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64477d2a-6239-4ecb-821d-c093c73716aa None admin] RBAC: Authorizing `identity:create_grant(project_id=1cc0eb0de66448aa8d924179a50e25e9, user_id=67cdfca3c1f54228aab73b276b6d34bb, role_id=7d473d5a7be94698b3b0ed9a2f499a56)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:41.833481 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64477d2a-6239-4ecb-821d-c093c73716aa None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:41.850075 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 74/148] 10.222.0.27 () {58 vars in 1439 bytes} [Sun Sep 11 00:38:41 2022] PUT /identity/v3/projects/1cc0eb0de66448aa8d924179a50e25e9/users/67cdfca3c1f54228aab73b276b6d34bb/roles/7d473d5a7be94698b3b0ed9a2f499a56 => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:38:42.787777 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd009b06-676a-4800-92d8-389618fa39b5 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:42.788516 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd009b06-676a-4800-92d8-389618fa39b5 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:42.788516 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd009b06-676a-4800-92d8-389618fa39b5 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:42.789733 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 75/149] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:38:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:38:42.803634 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd34d0dd-8594-4d41-88e5-c1f1ce917b1e None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:42.804239 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd34d0dd-8594-4d41-88e5-c1f1ce917b1e None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:42.804239 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd34d0dd-8594-4d41-88e5-c1f1ce917b1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:42.827839 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-fd34d0dd-8594-4d41-88e5-c1f1ce917b1e None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:38:42.829056 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-fd34d0dd-8594-4d41-88e5-c1f1ce917b1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:38:42.919207 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 75/150] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:38:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:38:43.372238 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db0ed63-8f52-491f-ada9-80842c0a8088 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:43.372909 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db0ed63-8f52-491f-ada9-80842c0a8088 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:43.373388 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db0ed63-8f52-491f-ada9-80842c0a8088 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:43.374961 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 76/151] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:38:43 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:38:43.392817 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e73686-edf4-4f13-8efe-19c4b73ef89c None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:43.393097 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e73686-edf4-4f13-8efe-19c4b73ef89c None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:43.393675 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e73686-edf4-4f13-8efe-19c4b73ef89c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:43.416971 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-91e73686-edf4-4f13-8efe-19c4b73ef89c None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:38:43.418411 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-91e73686-edf4-4f13-8efe-19c4b73ef89c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:38:43.486880 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5db0ed63-8f52-491f-ada9-80842c0a8088 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:43.520791 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 76/152] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:38:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:38:43.621549 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90265d82-ca63-4268-9ea1-abc907991bba None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:43.622091 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90265d82-ca63-4268-9ea1-abc907991bba None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:43.622641 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90265d82-ca63-4268-9ea1-abc907991bba None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:43.624364 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 77/153] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:38:43 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 11 00:38:43.630391 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea4b030-20ae-4163-8bf4-bdbc3ad117f5 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:43.631053 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea4b030-20ae-4163-8bf4-bdbc3ad117f5 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:43.631584 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea4b030-20ae-4163-8bf4-bdbc3ad117f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:43.651446 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-3ea4b030-20ae-4163-8bf4-bdbc3ad117f5 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:38:43.653108 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-3ea4b030-20ae-4163-8bf4-bdbc3ad117f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:38:43.660020 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5db0ed63-8f52-491f-ada9-80842c0a8088 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:43.682020 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:43.732722 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 78/154] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:38:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 11 00:38:43.751033 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddfe330d-b063-4c02-b767-b359257cd42a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:43.751690 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfe330d-b063-4c02-b767-b359257cd42a None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:43.751965 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfe330d-b063-4c02-b767-b359257cd42a None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:43.752235 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfe330d-b063-4c02-b767-b359257cd42a None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:43.771763 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddfe330d-b063-4c02-b767-b359257cd42a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:38:43.777908 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-ddfe330d-b063-4c02-b767-b359257cd42a None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 11 00:38:43.777908 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:38:43.777908 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:38:43.777908 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:38:43.777908 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:38:43.777908 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:38:43.777908 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:38:43.777908 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:38:43.777908 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:38:43.777908 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:38:43.777908 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:38:43.777908 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:38:43.777908 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:38:43.777908 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:38:43.777908 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:38:43.777908 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:38:43.778825 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:38:43.778825 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:38:43.778825 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:38:43.778825 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:38:43.778825 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:38:43.778825 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:38:43.778825 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:38:43.778825 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:38:43.778825 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:38:43.778825 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:38:43.778825 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:38:43.778825 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:38:43.778825 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:38:43.778825 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:38:43.778825 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:38:43.778825 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:38:43.779641 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:38:43.779641 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:38:43.779641 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:38:43.779641 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:38:43.779641 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:38:43.779641 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:38:43.779641 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:38:43.779641 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:38:43.779641 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:38:43.779641 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:38:43.779641 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 11 00:38:43.779641 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:38:43.781970 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 77/155] 10.222.0.27 () {56 vars in 1114 bytes} [Sun Sep 11 00:38:43 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 296 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:38:43.788010 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea4b030-20ae-4163-8bf4-bdbc3ad117f5 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:43.913787 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea4b030-20ae-4163-8bf4-bdbc3ad117f5 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:43.951093 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58605ba8-b355-42f8-b3fa-6d9e003d63f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:43.952063 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58605ba8-b355-42f8-b3fa-6d9e003d63f2 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:43.952551 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58605ba8-b355-42f8-b3fa-6d9e003d63f2 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:43.953024 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58605ba8-b355-42f8-b3fa-6d9e003d63f2 None admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:43.953955 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58605ba8-b355-42f8-b3fa-6d9e003d63f2 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:38:43.954459 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58605ba8-b355-42f8-b3fa-6d9e003d63f2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:43.956208 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58605ba8-b355-42f8-b3fa-6d9e003d63f2 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:43.964979 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 79/156] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:38:43 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:43.970727 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddfe330d-b063-4c02-b767-b359257cd42a None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:43.981440 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:44.096388 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddfe330d-b063-4c02-b767-b359257cd42a None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:44.142740 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9711d40f-c8be-4789-be7d-eb2d7acc9b9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:44.143758 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9711d40f-c8be-4789-be7d-eb2d7acc9b9f None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:44.144244 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9711d40f-c8be-4789-be7d-eb2d7acc9b9f None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:44.144729 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9711d40f-c8be-4789-be7d-eb2d7acc9b9f None admin] PATH_INFO: `/v3/users/2a97af87092c4282aa5985d5f454f26b` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:44.146114 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9711d40f-c8be-4789-be7d-eb2d7acc9b9f None admin] RBAC: Authorizing `identity:get_user(user_id=2a97af87092c4282aa5985d5f454f26b)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:44.147891 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9711d40f-c8be-4789-be7d-eb2d7acc9b9f None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:44.149714 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 78/157] 10.222.0.27 () {56 vars in 1177 bytes} [Sun Sep 11 00:38:43 2022] GET /identity/v3/users/2a97af87092c4282aa5985d5f454f26b => generated 275 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:44.156194 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58605ba8-b355-42f8-b3fa-6d9e003d63f2 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:44.185944 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:44.186826 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9711d40f-c8be-4789-be7d-eb2d7acc9b9f None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:44.323424 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58605ba8-b355-42f8-b3fa-6d9e003d63f2 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:44.349267 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:44.361867 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b48df611-087d-4834-9b84-60da1f04bbfc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:44.362658 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48df611-087d-4834-9b84-60da1f04bbfc None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:44.363119 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48df611-087d-4834-9b84-60da1f04bbfc None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:44.363491 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48df611-087d-4834-9b84-60da1f04bbfc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:44.401043 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9711d40f-c8be-4789-be7d-eb2d7acc9b9f None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:44.434701 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b48df611-087d-4834-9b84-60da1f04bbfc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:44.436286 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b48df611-087d-4834-9b84-60da1f04bbfc None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:44.467177 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:44.475516 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c10467f-a304-44b4-9089-908a406179c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:44.476201 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c10467f-a304-44b4-9089-908a406179c1 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:44.476443 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c10467f-a304-44b4-9089-908a406179c1 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:44.476715 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c10467f-a304-44b4-9089-908a406179c1 None admin] PATH_INFO: `/v3/roles/member` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:44.493402 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c10467f-a304-44b4-9089-908a406179c1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:38:44.507662 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-5c10467f-a304-44b4-9089-908a406179c1 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 11 00:38:44.507662 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:38:44.507662 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:38:44.507662 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:38:44.507662 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:38:44.507662 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:38:44.507662 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:38:44.507662 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:38:44.507662 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:38:44.507662 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:38:44.507662 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:38:44.507662 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:38:44.507662 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:38:44.507662 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:38:44.507662 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:38:44.507662 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:38:44.508658 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:38:44.508658 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:38:44.508658 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:38:44.508658 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:38:44.508658 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:38:44.508658 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:38:44.508658 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:38:44.508658 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:38:44.508658 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:38:44.508658 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:38:44.508658 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:38:44.508658 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:38:44.508658 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:38:44.508658 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:38:44.508658 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:38:44.508658 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:38:44.510027 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:38:44.510027 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:38:44.510027 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:38:44.510027 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:38:44.510027 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:38:44.510027 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:38:44.510027 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:38:44.510027 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:38:44.510027 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:38:44.510027 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:38:44.510027 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 11 00:38:44.510027 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:38:44.510027 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 79/158] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:38:44 2022] GET /identity/v3/roles/member => generated 84 bytes in 323 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:38:44.511089 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c10467f-a304-44b4-9089-908a406179c1 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:44.607768 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 80/159] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:38:44 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 452 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 11 00:38:44.608912 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b48df611-087d-4834-9b84-60da1f04bbfc None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:44.682667 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c10467f-a304-44b4-9089-908a406179c1 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:44.747472 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bae57b8-4d01-462d-8bd4-36b31c22a329 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:44.750237 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bae57b8-4d01-462d-8bd4-36b31c22a329 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:44.750237 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bae57b8-4d01-462d-8bd4-36b31c22a329 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:44.750237 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bae57b8-4d01-462d-8bd4-36b31c22a329 None admin] PATH_INFO: `/v3/roles/member` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:44.755100 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:44.765848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bae57b8-4d01-462d-8bd4-36b31c22a329 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:38:44.770817 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-8bae57b8-4d01-462d-8bd4-36b31c22a329 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 11 00:38:44.770817 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:38:44.770817 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:38:44.770817 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:38:44.770817 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:38:44.770817 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:38:44.770817 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:38:44.770817 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:38:44.770817 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:38:44.770817 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:38:44.770817 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:38:44.770817 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:38:44.770817 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:38:44.770817 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:38:44.770817 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:38:44.770817 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:38:44.771748 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:38:44.771748 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:38:44.771748 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:38:44.771748 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:38:44.771748 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:38:44.771748 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:38:44.771748 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:38:44.771748 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:38:44.771748 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:38:44.771748 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:38:44.771748 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:38:44.771748 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:38:44.771748 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:38:44.771748 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:38:44.771748 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:38:44.771748 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:38:44.772544 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:38:44.772544 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:38:44.772544 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:38:44.772544 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:38:44.772544 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:38:44.772544 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:38:44.772544 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:38:44.772544 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:38:44.772544 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:38:44.772544 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:38:44.772544 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 11 00:38:44.772544 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:38:44.780191 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 80/160] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:38:44 2022] GET /identity/v3/roles/member => generated 84 bytes in 271 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:38:44.781153 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bae57b8-4d01-462d-8bd4-36b31c22a329 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:44.854309 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b48df611-087d-4834-9b84-60da1f04bbfc None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:44.899129 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:44.928203 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9cdfa2a-87bb-4197-88f1-5ce62ad44058 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:44.929062 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9cdfa2a-87bb-4197-88f1-5ce62ad44058 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:44.929435 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9cdfa2a-87bb-4197-88f1-5ce62ad44058 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:44.929708 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9cdfa2a-87bb-4197-88f1-5ce62ad44058 None admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:44.930482 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9cdfa2a-87bb-4197-88f1-5ce62ad44058 None admin] RBAC: Adding query filter params (name=member) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:38:44.930822 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9cdfa2a-87bb-4197-88f1-5ce62ad44058 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:44.932321 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9cdfa2a-87bb-4197-88f1-5ce62ad44058 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:44.935949 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bae57b8-4d01-462d-8bd4-36b31c22a329 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:44.953824 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 81/161] 10.222.0.27 () {56 vars in 1101 bytes} [Sun Sep 11 00:38:44 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 345 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:44.954576 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9cdfa2a-87bb-4197-88f1-5ce62ad44058 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:45.034904 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3431b8cd-8903-4339-9bca-4b91f4d11fce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:45.035798 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3431b8cd-8903-4339-9bca-4b91f4d11fce None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:45.036057 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3431b8cd-8903-4339-9bca-4b91f4d11fce None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:45.036328 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3431b8cd-8903-4339-9bca-4b91f4d11fce None admin] PATH_INFO: `/v3/projects/1cc0eb0de66448aa8d924179a50e25e9` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:45.039813 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3431b8cd-8903-4339-9bca-4b91f4d11fce None admin] RBAC: Authorizing `identity:get_project(project_id=1cc0eb0de66448aa8d924179a50e25e9)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:45.041976 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3431b8cd-8903-4339-9bca-4b91f4d11fce None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:45.044068 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 81/162] 10.222.0.27 () {56 vars in 1186 bytes} [Sun Sep 11 00:38:44 2022] GET /identity/v3/projects/1cc0eb0de66448aa8d924179a50e25e9 => generated 298 bytes in 263 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:45.044916 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3431b8cd-8903-4339-9bca-4b91f4d11fce None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:45.100081 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9cdfa2a-87bb-4197-88f1-5ce62ad44058 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:45.192933 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98145f13-6cee-4e5f-988b-8f4f923d6acb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:45.193545 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98145f13-6cee-4e5f-988b-8f4f923d6acb None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:45.193823 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98145f13-6cee-4e5f-988b-8f4f923d6acb None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:45.194092 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98145f13-6cee-4e5f-988b-8f4f923d6acb None admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:45.194763 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98145f13-6cee-4e5f-988b-8f4f923d6acb None admin] RBAC: Adding query filter params (name=member) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:38:45.195093 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98145f13-6cee-4e5f-988b-8f4f923d6acb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:45.196469 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98145f13-6cee-4e5f-988b-8f4f923d6acb None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:45.214902 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 82/163] 10.222.0.27 () {56 vars in 1101 bytes} [Sun Sep 11 00:38:44 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:45.216156 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2dc3a73-ca14-48ca-8776-ee051115f733 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:45.216441 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2dc3a73-ca14-48ca-8776-ee051115f733 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:45.216714 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2dc3a73-ca14-48ca-8776-ee051115f733 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:45.217928 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 83/164] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:38:45 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:38:45.218619 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2dc3a73-ca14-48ca-8776-ee051115f733 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:45.227127 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:45.288849 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3431b8cd-8903-4339-9bca-4b91f4d11fce None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:45.331189 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:45.356520 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e81edc91-8d24-4f57-a8a1-788e7b1aea76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:45.357169 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81edc91-8d24-4f57-a8a1-788e7b1aea76 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:45.357445 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81edc91-8d24-4f57-a8a1-788e7b1aea76 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:45.357713 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81edc91-8d24-4f57-a8a1-788e7b1aea76 None admin] PATH_INFO: `/v3/users/2a97af87092c4282aa5985d5f454f26b` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:45.359233 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e81edc91-8d24-4f57-a8a1-788e7b1aea76 None admin] RBAC: Authorizing `identity:get_user(user_id=2a97af87092c4282aa5985d5f454f26b)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:45.360700 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e81edc91-8d24-4f57-a8a1-788e7b1aea76 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:45.362439 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 82/165] 10.222.0.27 () {56 vars in 1177 bytes} [Sun Sep 11 00:38:45 2022] GET /identity/v3/users/2a97af87092c4282aa5985d5f454f26b => generated 275 bytes in 318 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:45.368195 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de5d660-bdad-4bd7-84a4-ed827a823f1d None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:45.368471 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de5d660-bdad-4bd7-84a4-ed827a823f1d None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:45.369303 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de5d660-bdad-4bd7-84a4-ed827a823f1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:45.400796 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-3de5d660-bdad-4bd7-84a4-ed827a823f1d None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:38:45.402098 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-3de5d660-bdad-4bd7-84a4-ed827a823f1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:38:45.421417 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2dc3a73-ca14-48ca-8776-ee051115f733 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:45.488690 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f0fdf1e-3fa3-4576-8357-6bd08c660753 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:45.489305 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0fdf1e-3fa3-4576-8357-6bd08c660753 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:45.489581 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0fdf1e-3fa3-4576-8357-6bd08c660753 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:45.489848 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0fdf1e-3fa3-4576-8357-6bd08c660753 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:45.490588 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f0fdf1e-3fa3-4576-8357-6bd08c660753 None admin] RBAC: Adding query filter params (role.id=518eab0f758241369edcdd997b98ad12, scope.project.id=1cc0eb0de66448aa8d924179a50e25e9, user.id=2a97af87092c4282aa5985d5f454f26b) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:38:45.490917 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f0fdf1e-3fa3-4576-8357-6bd08c660753 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:45.492364 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f0fdf1e-3fa3-4576-8357-6bd08c660753 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:45.504036 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 84/166] 10.222.0.27 () {56 vars in 1374 bytes} [Sun Sep 11 00:38:45 2022] GET /identity/v3/role_assignments?user.id=2a97af87092c4282aa5985d5f454f26b&scope.project.id=1cc0eb0de66448aa8d924179a50e25e9&role.id=518eab0f758241369edcdd997b98ad12 => generated 608 bytes in 285 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:45.504688 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f0fdf1e-3fa3-4576-8357-6bd08c660753 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:45.543404 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 83/167] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:38:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:38:45.544020 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de5d660-bdad-4bd7-84a4-ed827a823f1d None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:45.649276 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f0fdf1e-3fa3-4576-8357-6bd08c660753 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:45.666813 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:45.667206 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:45.718839 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de5d660-bdad-4bd7-84a4-ed827a823f1d None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:45.723562 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-125b1bcb-665f-41cd-8f2f-a3002fc40ba8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:45.724216 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125b1bcb-665f-41cd-8f2f-a3002fc40ba8 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:45.724496 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125b1bcb-665f-41cd-8f2f-a3002fc40ba8 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:45.724768 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125b1bcb-665f-41cd-8f2f-a3002fc40ba8 None admin] PATH_INFO: `/v3/users/2a97af87092c4282aa5985d5f454f26b` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:45.729496 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-125b1bcb-665f-41cd-8f2f-a3002fc40ba8 None admin] RBAC: Authorizing `identity:get_user(user_id=2a97af87092c4282aa5985d5f454f26b)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:45.731018 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-125b1bcb-665f-41cd-8f2f-a3002fc40ba8 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:45.732998 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 85/168] 10.222.0.27 () {56 vars in 1177 bytes} [Sun Sep 11 00:38:45 2022] GET /identity/v3/users/2a97af87092c4282aa5985d5f454f26b => generated 275 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:45.738480 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-125b1bcb-665f-41cd-8f2f-a3002fc40ba8 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:45.766004 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9389b136-e0c4-4614-8775-187bf105d2a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:45.767030 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9389b136-e0c4-4614-8775-187bf105d2a5 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:45.767523 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9389b136-e0c4-4614-8775-187bf105d2a5 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:45.768002 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9389b136-e0c4-4614-8775-187bf105d2a5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:45.902764 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9389b136-e0c4-4614-8775-187bf105d2a5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:45.904200 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9389b136-e0c4-4614-8775-187bf105d2a5 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:45.921681 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-125b1bcb-665f-41cd-8f2f-a3002fc40ba8 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:45.960678 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e9baae-b3f1-4b75-bb66-92eb7acbbab9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:45.961313 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e9baae-b3f1-4b75-bb66-92eb7acbbab9 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:45.962326 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e9baae-b3f1-4b75-bb66-92eb7acbbab9 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:45.963586 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e9baae-b3f1-4b75-bb66-92eb7acbbab9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:46.035504 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:46.088776 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:46.098225 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 84/169] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:38:45 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 555 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 11 00:38:46.099711 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-734f6e6a-95c0-4490-b52a-8e98d16ec490 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:46.100083 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-734f6e6a-95c0-4490-b52a-8e98d16ec490 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:46.100451 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-734f6e6a-95c0-4490-b52a-8e98d16ec490 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:46.101924 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 85/170] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:38:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:38:46.103962 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-734f6e6a-95c0-4490-b52a-8e98d16ec490 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:46.127344 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20e9baae-b3f1-4b75-bb66-92eb7acbbab9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:46.129218 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20e9baae-b3f1-4b75-bb66-92eb7acbbab9 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:46.258403 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-734f6e6a-95c0-4490-b52a-8e98d16ec490 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:46.275592 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 86/171] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:38:45 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 538 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 11 00:38:46.277288 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e9baae-b3f1-4b75-bb66-92eb7acbbab9 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:46.300295 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d8ea77-5911-4a97-8c76-7b0a2bfb3260 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:46.300980 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d8ea77-5911-4a97-8c76-7b0a2bfb3260 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:46.301260 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d8ea77-5911-4a97-8c76-7b0a2bfb3260 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:46.301525 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d8ea77-5911-4a97-8c76-7b0a2bfb3260 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:46.313060 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91d8ea77-5911-4a97-8c76-7b0a2bfb3260 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:38:46.317955 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-91d8ea77-5911-4a97-8c76-7b0a2bfb3260 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:38:46.317955 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:38:46.317955 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:38:46.317955 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:38:46.317955 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:38:46.317955 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:38:46.317955 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:38:46.317955 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:38:46.317955 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:38:46.317955 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:38:46.317955 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:38:46.317955 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:38:46.317955 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:38:46.317955 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:38:46.317955 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:38:46.317955 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:38:46.319062 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:38:46.319062 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:38:46.319062 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:38:46.319062 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:38:46.319062 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:38:46.319062 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:38:46.319062 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:38:46.319062 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:38:46.319062 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:38:46.319062 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:38:46.319062 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:38:46.319062 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:38:46.319062 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:38:46.319062 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:38:46.319062 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:38:46.319062 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:38:46.320174 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:38:46.320174 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:38:46.320174 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:38:46.320174 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:38:46.320174 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:38:46.320174 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:38:46.320174 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:38:46.320174 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:38:46.320174 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:38:46.320174 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:38:46.320174 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:38:46.320174 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:38:46.320174 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 86/172] 10.222.0.27 () {56 vars in 1096 bytes} [Sun Sep 11 00:38:46 2022] GET /identity/v3/roles/admin => generated 83 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:38:46.320922 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646adb8b-d5de-4873-9dfb-552c5de3097d None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:46.320922 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646adb8b-d5de-4873-9dfb-552c5de3097d None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:46.321061 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646adb8b-d5de-4873-9dfb-552c5de3097d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:46.340094 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-646adb8b-d5de-4873-9dfb-552c5de3097d None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:38:46.341333 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-646adb8b-d5de-4873-9dfb-552c5de3097d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:38:46.358303 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:46.380893 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:46.407412 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e9baae-b3f1-4b75-bb66-92eb7acbbab9 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:46.417586 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 87/173] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:38:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:38:46.428087 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4355ab6f-8308-4f2f-8df3-6f52eb3c1203 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:46.438848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4355ab6f-8308-4f2f-8df3-6f52eb3c1203 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:46.438848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4355ab6f-8308-4f2f-8df3-6f52eb3c1203 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:46.438848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 88/174] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:38:46 2022] GET /identity => generated 270 bytes in 12 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:38:46.438848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4355ab6f-8308-4f2f-8df3-6f52eb3c1203 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:46.461852 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2b40055-22b0-42d7-a8c6-fed86ba70166 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:46.462647 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b40055-22b0-42d7-a8c6-fed86ba70166 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:46.462976 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b40055-22b0-42d7-a8c6-fed86ba70166 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:46.463247 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b40055-22b0-42d7-a8c6-fed86ba70166 None admin] PATH_INFO: `/v3/projects/1cc0eb0de66448aa8d924179a50e25e9` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:46.465560 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2b40055-22b0-42d7-a8c6-fed86ba70166 None admin] RBAC: Authorizing `identity:get_project(project_id=1cc0eb0de66448aa8d924179a50e25e9)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:46.467031 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2b40055-22b0-42d7-a8c6-fed86ba70166 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:46.468882 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 87/175] 10.222.0.27 () {56 vars in 1186 bytes} [Sun Sep 11 00:38:46 2022] GET /identity/v3/projects/1cc0eb0de66448aa8d924179a50e25e9 => generated 298 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:46.469796 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126fa380-7015-4b6a-b69a-f1433e51302e None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:46.470090 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126fa380-7015-4b6a-b69a-f1433e51302e None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:46.470359 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126fa380-7015-4b6a-b69a-f1433e51302e None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:46.476556 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 88/176] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:38:46 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:38:46.476556 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-126fa380-7015-4b6a-b69a-f1433e51302e None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:46.572483 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4355ab6f-8308-4f2f-8df3-6f52eb3c1203 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:46.589618 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-126fa380-7015-4b6a-b69a-f1433e51302e None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:46.606640 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d661fe3-de54-4d57-ae16-e0cc12b9efd1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:46.607308 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d661fe3-de54-4d57-ae16-e0cc12b9efd1 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:46.607588 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d661fe3-de54-4d57-ae16-e0cc12b9efd1 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:46.607862 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d661fe3-de54-4d57-ae16-e0cc12b9efd1 None admin] PATH_INFO: `/v3/projects/6b1547a650214e0dbd243c050595fb86` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:46.608984 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d661fe3-de54-4d57-ae16-e0cc12b9efd1 None admin] RBAC: Authorizing `identity:get_project(project_id=6b1547a650214e0dbd243c050595fb86)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:46.610451 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d661fe3-de54-4d57-ae16-e0cc12b9efd1 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:46.612190 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 89/177] 10.222.0.27 () {56 vars in 1186 bytes} [Sun Sep 11 00:38:46 2022] GET /identity/v3/projects/6b1547a650214e0dbd243c050595fb86 => generated 312 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:46.613363 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a58cacb-015d-476f-8f36-887590212f08 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:46.613659 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a58cacb-015d-476f-8f36-887590212f08 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:46.613919 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a58cacb-015d-476f-8f36-887590212f08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:46.630548 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06cfad75-4074-4cb3-9af5-dc4b30e84c68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:46.631202 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06cfad75-4074-4cb3-9af5-dc4b30e84c68 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:46.631479 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06cfad75-4074-4cb3-9af5-dc4b30e84c68 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:46.631744 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06cfad75-4074-4cb3-9af5-dc4b30e84c68 None admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:46.632401 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06cfad75-4074-4cb3-9af5-dc4b30e84c68 None admin] RBAC: Adding query filter params (name=admin) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:38:46.632569 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-4a58cacb-015d-476f-8f36-887590212f08 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:38:46.632737 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06cfad75-4074-4cb3-9af5-dc4b30e84c68 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:46.633702 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-4a58cacb-015d-476f-8f36-887590212f08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:38:46.634054 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06cfad75-4074-4cb3-9af5-dc4b30e84c68 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:46.647649 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 89/178] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:38:46 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:46.647649 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06cfad75-4074-4cb3-9af5-dc4b30e84c68 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:46.652932 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:46.658620 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:46.711521 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 90/179] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:38:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:38:46.713283 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2cf077-e205-4844-85cb-8c9ec1b56031 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:46.713559 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2cf077-e205-4844-85cb-8c9ec1b56031 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:46.713827 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2cf077-e205-4844-85cb-8c9ec1b56031 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:46.731611 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-ba2cf077-e205-4844-85cb-8c9ec1b56031 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:38:46.732785 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-ba2cf077-e205-4844-85cb-8c9ec1b56031 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:38:46.757875 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06cfad75-4074-4cb3-9af5-dc4b30e84c68 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:46.796388 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a96f0dbb-0fba-4d41-8a55-96430803cfc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:46.797304 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96f0dbb-0fba-4d41-8a55-96430803cfc1 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:46.797742 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96f0dbb-0fba-4d41-8a55-96430803cfc1 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:46.798025 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96f0dbb-0fba-4d41-8a55-96430803cfc1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:46.798767 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a96f0dbb-0fba-4d41-8a55-96430803cfc1 None admin] RBAC: Adding query filter params (role.id=2e64fc597b6447df99a7992d411d1a34, scope.project.id=1cc0eb0de66448aa8d924179a50e25e9, user.id=2a97af87092c4282aa5985d5f454f26b) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:38:46.799106 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a96f0dbb-0fba-4d41-8a55-96430803cfc1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:46.800550 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a96f0dbb-0fba-4d41-8a55-96430803cfc1 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:46.801751 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 91/180] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:38:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:38:46.814186 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 90/181] 10.222.0.27 () {56 vars in 1374 bytes} [Sun Sep 11 00:38:46 2022] GET /identity/v3/role_assignments?user.id=2a97af87092c4282aa5985d5f454f26b&scope.project.id=1cc0eb0de66448aa8d924179a50e25e9&role.id=2e64fc597b6447df99a7992d411d1a34 => generated 608 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:46.814186 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42acc27-ea96-431f-8449-35343a6b5335 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:46.814186 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42acc27-ea96-431f-8449-35343a6b5335 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:46.814186 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42acc27-ea96-431f-8449-35343a6b5335 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:46.814594 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2cf077-e205-4844-85cb-8c9ec1b56031 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:46.815080 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 91/182] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:38:46 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:38:46.816375 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42acc27-ea96-431f-8449-35343a6b5335 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:46.953923 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:46.971744 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:46.998776 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2cf077-e205-4844-85cb-8c9ec1b56031 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:47.011553 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42acc27-ea96-431f-8449-35343a6b5335 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:47.051160 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b60faa1-b30f-45fc-84f9-7b15ab9c3da7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:47.052405 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b60faa1-b30f-45fc-84f9-7b15ab9c3da7 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:47.052857 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b60faa1-b30f-45fc-84f9-7b15ab9c3da7 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:47.053293 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b60faa1-b30f-45fc-84f9-7b15ab9c3da7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:47.054200 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b60faa1-b30f-45fc-84f9-7b15ab9c3da7 None admin] RBAC: Adding query filter params (role.id=2e64fc597b6447df99a7992d411d1a34, scope.project.id=6b1547a650214e0dbd243c050595fb86, user.id=2a97af87092c4282aa5985d5f454f26b) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:38:47.054505 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b60faa1-b30f-45fc-84f9-7b15ab9c3da7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:47.055028 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f38ac3-a298-4ca3-b360-50ae2649e18a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:47.055668 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f38ac3-a298-4ca3-b360-50ae2649e18a None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:47.055937 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f38ac3-a298-4ca3-b360-50ae2649e18a None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:47.056084 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b60faa1-b30f-45fc-84f9-7b15ab9c3da7 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:47.056333 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f38ac3-a298-4ca3-b360-50ae2649e18a None admin] PATH_INFO: `/v3/users/36edaa883d654921ac17e7745c0ff320` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:47.057595 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1f38ac3-a298-4ca3-b360-50ae2649e18a None admin] RBAC: Authorizing `identity:get_user(user_id=36edaa883d654921ac17e7745c0ff320)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:47.059179 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1f38ac3-a298-4ca3-b360-50ae2649e18a None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:47.060880 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 92/183] 10.222.0.27 () {56 vars in 1177 bytes} [Sun Sep 11 00:38:46 2022] GET /identity/v3/users/36edaa883d654921ac17e7745c0ff320 => generated 247 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:47.063143 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 92/184] 10.222.0.27 () {56 vars in 1374 bytes} [Sun Sep 11 00:38:46 2022] GET /identity/v3/role_assignments?user.id=2a97af87092c4282aa5985d5f454f26b&scope.project.id=6b1547a650214e0dbd243c050595fb86&role.id=2e64fc597b6447df99a7992d411d1a34 => generated 608 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:47.064578 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01db071f-277f-4cef-9182-15178ac1dad8 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:47.065053 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01db071f-277f-4cef-9182-15178ac1dad8 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:47.065320 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01db071f-277f-4cef-9182-15178ac1dad8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:47.066655 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f38ac3-a298-4ca3-b360-50ae2649e18a None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:47.089330 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-01db071f-277f-4cef-9182-15178ac1dad8 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:38:47.090589 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-01db071f-277f-4cef-9182-15178ac1dad8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:38:47.190291 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 93/185] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:38:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:38:47.191588 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01db071f-277f-4cef-9182-15178ac1dad8 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:47.220459 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f38ac3-a298-4ca3-b360-50ae2649e18a None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:47.265415 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b134ee82-b7de-45aa-8586-dd9b7d692460 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:47.266059 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b134ee82-b7de-45aa-8586-dd9b7d692460 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:47.266588 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b134ee82-b7de-45aa-8586-dd9b7d692460 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:47.266886 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b134ee82-b7de-45aa-8586-dd9b7d692460 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:47.275990 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:47.279612 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b134ee82-b7de-45aa-8586-dd9b7d692460 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:38:47.279955 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:47.285589 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-b134ee82-b7de-45aa-8586-dd9b7d692460 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:38:47.285589 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:38:47.285589 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:38:47.285589 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:38:47.285589 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:38:47.285589 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:38:47.285589 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:38:47.285589 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:38:47.285589 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:38:47.285589 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:38:47.285589 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:38:47.285589 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:38:47.285589 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:38:47.285589 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:38:47.285589 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:38:47.285589 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:38:47.286581 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:38:47.286581 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:38:47.286581 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:38:47.286581 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:38:47.286581 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:38:47.286581 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:38:47.286581 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:38:47.286581 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:38:47.286581 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:38:47.286581 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:38:47.286581 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:38:47.286581 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:38:47.286581 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:38:47.286581 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:38:47.286581 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:38:47.286581 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:38:47.287436 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:38:47.287436 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:38:47.287436 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:38:47.287436 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:38:47.287436 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:38:47.287436 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:38:47.287436 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:38:47.287436 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:38:47.287436 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:38:47.287436 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:38:47.287436 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:38:47.287436 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:38:47.290408 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 93/186] 10.222.0.27 () {56 vars in 1096 bytes} [Sun Sep 11 00:38:47 2022] GET /identity/v3/roles/admin => generated 83 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:38:47.295358 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b134ee82-b7de-45aa-8586-dd9b7d692460 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:47.321625 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01db071f-277f-4cef-9182-15178ac1dad8 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:47.362831 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3943ae30-5b27-4a49-8c30-fefe9debcec2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:47.363969 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3943ae30-5b27-4a49-8c30-fefe9debcec2 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:47.364477 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3943ae30-5b27-4a49-8c30-fefe9debcec2 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:47.364948 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3943ae30-5b27-4a49-8c30-fefe9debcec2 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:47.377264 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3943ae30-5b27-4a49-8c30-fefe9debcec2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:38:47.382706 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-3943ae30-5b27-4a49-8c30-fefe9debcec2 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 11 00:38:47.382706 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:38:47.382706 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:38:47.382706 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:38:47.382706 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:38:47.382706 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:38:47.382706 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:38:47.382706 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:38:47.382706 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:38:47.382706 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:38:47.382706 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:38:47.382706 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:38:47.382706 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:38:47.382706 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:38:47.382706 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:38:47.382706 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:38:47.383664 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:38:47.383664 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:38:47.383664 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:38:47.383664 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:38:47.383664 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:38:47.383664 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:38:47.383664 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:38:47.383664 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:38:47.383664 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:38:47.383664 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:38:47.383664 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:38:47.383664 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:38:47.383664 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:38:47.383664 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:38:47.383664 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:38:47.383664 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:38:47.385145 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:38:47.385145 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:38:47.385145 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:38:47.385145 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:38:47.385145 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:38:47.385145 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:38:47.385145 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:38:47.385145 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:38:47.385145 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:38:47.385145 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:38:47.385145 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 11 00:38:47.385145 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:38:47.386032 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 94/187] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:38:47 2022] GET /identity/v3/roles/reader => generated 84 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:38:47.387179 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3943ae30-5b27-4a49-8c30-fefe9debcec2 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:47.411206 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b134ee82-b7de-45aa-8586-dd9b7d692460 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:47.448892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e629dd-b54c-41b8-90e9-23eb242fe187 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:47.449627 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e629dd-b54c-41b8-90e9-23eb242fe187 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:47.449905 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e629dd-b54c-41b8-90e9-23eb242fe187 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:47.450181 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e629dd-b54c-41b8-90e9-23eb242fe187 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:47.493626 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3943ae30-5b27-4a49-8c30-fefe9debcec2 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:47.523137 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:47.523453 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:47.526571 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8e629dd-b54c-41b8-90e9-23eb242fe187 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:47.528681 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8e629dd-b54c-41b8-90e9-23eb242fe187 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:47.529097 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd9c6994-b8cf-4f9a-bc02-6fcbd3149f6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:47.529720 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9c6994-b8cf-4f9a-bc02-6fcbd3149f6b None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:47.529995 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9c6994-b8cf-4f9a-bc02-6fcbd3149f6b None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:47.530264 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9c6994-b8cf-4f9a-bc02-6fcbd3149f6b None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:47.541188 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd9c6994-b8cf-4f9a-bc02-6fcbd3149f6b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:38:47.546499 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-dd9c6994-b8cf-4f9a-bc02-6fcbd3149f6b None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 11 00:38:47.546499 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:38:47.546499 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:38:47.546499 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:38:47.546499 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:38:47.546499 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:38:47.546499 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:38:47.546499 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:38:47.546499 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:38:47.546499 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:38:47.546499 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:38:47.546499 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:38:47.546499 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:38:47.546499 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:38:47.546499 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:38:47.546499 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:38:47.547566 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:38:47.547566 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:38:47.547566 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:38:47.547566 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:38:47.547566 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:38:47.547566 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:38:47.547566 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:38:47.547566 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:38:47.547566 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:38:47.547566 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:38:47.547566 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:38:47.547566 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:38:47.547566 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:38:47.547566 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:38:47.547566 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:38:47.547566 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:38:47.548378 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:38:47.548378 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:38:47.548378 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:38:47.548378 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:38:47.548378 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:38:47.548378 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:38:47.548378 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:38:47.548378 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:38:47.548378 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:38:47.548378 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:38:47.548378 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 11 00:38:47.548378 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:38:47.548378 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 95/188] 10.222.0.27 () {56 vars in 1114 bytes} [Sun Sep 11 00:38:47 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:38:47.549018 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd9c6994-b8cf-4f9a-bc02-6fcbd3149f6b None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:47.629865 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 94/189] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:38:47 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 339 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 11 00:38:47.631073 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e629dd-b54c-41b8-90e9-23eb242fe187 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:47.646092 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd9c6994-b8cf-4f9a-bc02-6fcbd3149f6b None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:47.679961 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaa7e135-7019-41a1-89a4-e9c1001b09c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:47.680557 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa7e135-7019-41a1-89a4-e9c1001b09c9 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:47.680831 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa7e135-7019-41a1-89a4-e9c1001b09c9 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:47.681161 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa7e135-7019-41a1-89a4-e9c1001b09c9 None admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:47.681820 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaa7e135-7019-41a1-89a4-e9c1001b09c9 None admin] RBAC: Adding query filter params (name=admin) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:38:47.682117 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaa7e135-7019-41a1-89a4-e9c1001b09c9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:47.683537 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaa7e135-7019-41a1-89a4-e9c1001b09c9 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:47.697831 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 96/190] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:38:47 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:47.697831 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaa7e135-7019-41a1-89a4-e9c1001b09c9 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:47.734040 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:47.737232 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e629dd-b54c-41b8-90e9-23eb242fe187 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:47.748187 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:47.773791 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d857c4b-c279-48c2-862a-280564cae915 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:47.774478 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d857c4b-c279-48c2-862a-280564cae915 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:47.775452 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d857c4b-c279-48c2-862a-280564cae915 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:47.776011 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d857c4b-c279-48c2-862a-280564cae915 None admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:47.776677 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d857c4b-c279-48c2-862a-280564cae915 None admin] RBAC: Adding query filter params (name=reader) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:38:47.776979 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d857c4b-c279-48c2-862a-280564cae915 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:47.778337 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d857c4b-c279-48c2-862a-280564cae915 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:47.786429 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 95/191] 10.222.0.27 () {56 vars in 1101 bytes} [Sun Sep 11 00:38:47 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:47.787389 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d857c4b-c279-48c2-862a-280564cae915 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:47.793792 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaa7e135-7019-41a1-89a4-e9c1001b09c9 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:47.827843 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89cc9aa0-d4a3-4789-bf0a-c7af8602e423 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:47.828461 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89cc9aa0-d4a3-4789-bf0a-c7af8602e423 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:47.828735 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89cc9aa0-d4a3-4789-bf0a-c7af8602e423 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:47.829003 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89cc9aa0-d4a3-4789-bf0a-c7af8602e423 None admin] PATH_INFO: `/v3/domains/default` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:47.830097 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89cc9aa0-d4a3-4789-bf0a-c7af8602e423 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:47.831561 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89cc9aa0-d4a3-4789-bf0a-c7af8602e423 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:47.833083 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 97/192] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:38:47 2022] GET /identity/v3/domains/default => generated 199 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:47.835886 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89cc9aa0-d4a3-4789-bf0a-c7af8602e423 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:47.890732 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d857c4b-c279-48c2-862a-280564cae915 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:47.924188 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f4b9c49-e1f6-4548-b5eb-9763b4f55a62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:47.924798 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4b9c49-e1f6-4548-b5eb-9763b4f55a62 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:47.925238 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4b9c49-e1f6-4548-b5eb-9763b4f55a62 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:47.925516 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4b9c49-e1f6-4548-b5eb-9763b4f55a62 None admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:47.926187 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f4b9c49-e1f6-4548-b5eb-9763b4f55a62 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:38:47.926493 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f4b9c49-e1f6-4548-b5eb-9763b4f55a62 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:47.927978 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f4b9c49-e1f6-4548-b5eb-9763b4f55a62 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:47.935837 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89cc9aa0-d4a3-4789-bf0a-c7af8602e423 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:47.936335 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 96/193] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:38:47 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:47.937147 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f4b9c49-e1f6-4548-b5eb-9763b4f55a62 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:47.962976 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:47.963139 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:47.975818 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aeb96e2-f4bb-47f6-877b-3ef13372c216 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:47.976443 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aeb96e2-f4bb-47f6-877b-3ef13372c216 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:47.976720 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aeb96e2-f4bb-47f6-877b-3ef13372c216 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:47.976990 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aeb96e2-f4bb-47f6-877b-3ef13372c216 None admin] PATH_INFO: `/v3/projects/1cc0eb0de66448aa8d924179a50e25e9` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:47.978981 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aeb96e2-f4bb-47f6-877b-3ef13372c216 None admin] RBAC: Authorizing `identity:get_project(project_id=1cc0eb0de66448aa8d924179a50e25e9)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:47.980434 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aeb96e2-f4bb-47f6-877b-3ef13372c216 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:47.982050 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 98/194] 10.222.0.27 () {56 vars in 1186 bytes} [Sun Sep 11 00:38:47 2022] GET /identity/v3/projects/1cc0eb0de66448aa8d924179a50e25e9 => generated 298 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:47.982910 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aeb96e2-f4bb-47f6-877b-3ef13372c216 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:48.043361 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f4b9c49-e1f6-4548-b5eb-9763b4f55a62 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:48.075297 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373ddd93-fe73-4ff0-a88f-437dfd5d9022 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:48.075924 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373ddd93-fe73-4ff0-a88f-437dfd5d9022 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:48.078644 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373ddd93-fe73-4ff0-a88f-437dfd5d9022 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:48.078644 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373ddd93-fe73-4ff0-a88f-437dfd5d9022 None admin] PATH_INFO: `/v3/users/f4b18ce48c9e4f2196e1fa77aec45f6f` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:48.080324 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aeb96e2-f4bb-47f6-877b-3ef13372c216 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:48.092311 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-373ddd93-fe73-4ff0-a88f-437dfd5d9022 None admin] RBAC: Authorizing `identity:get_user(user_id=f4b18ce48c9e4f2196e1fa77aec45f6f)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:48.093813 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-373ddd93-fe73-4ff0-a88f-437dfd5d9022 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:48.095495 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 97/195] 10.222.0.27 () {56 vars in 1177 bytes} [Sun Sep 11 00:38:47 2022] GET /identity/v3/users/f4b18ce48c9e4f2196e1fa77aec45f6f => generated 283 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:48.096234 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373ddd93-fe73-4ff0-a88f-437dfd5d9022 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:48.120226 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b5f77e-3fde-483a-93e4-efd6e8a7f14e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:48.120851 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b5f77e-3fde-483a-93e4-efd6e8a7f14e None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:48.121121 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b5f77e-3fde-483a-93e4-efd6e8a7f14e None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:48.121404 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b5f77e-3fde-483a-93e4-efd6e8a7f14e None admin] PATH_INFO: `/v3/users/67cdfca3c1f54228aab73b276b6d34bb` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:48.122518 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b5f77e-3fde-483a-93e4-efd6e8a7f14e None admin] RBAC: Authorizing `identity:get_user(user_id=67cdfca3c1f54228aab73b276b6d34bb)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:48.124002 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b5f77e-3fde-483a-93e4-efd6e8a7f14e None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:48.125548 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 99/196] 10.222.0.27 () {56 vars in 1177 bytes} [Sun Sep 11 00:38:47 2022] GET /identity/v3/users/67cdfca3c1f54228aab73b276b6d34bb => generated 289 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:48.127946 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b5f77e-3fde-483a-93e4-efd6e8a7f14e None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:48.177387 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:48.185180 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:48.195725 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373ddd93-fe73-4ff0-a88f-437dfd5d9022 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:48.226924 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b5f77e-3fde-483a-93e4-efd6e8a7f14e None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:48.229906 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fca545b-b330-4aa9-82d0-566ad0ac2c8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:48.230507 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fca545b-b330-4aa9-82d0-566ad0ac2c8c None admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:48.230807 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fca545b-b330-4aa9-82d0-566ad0ac2c8c None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:48.231078 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fca545b-b330-4aa9-82d0-566ad0ac2c8c None admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:48.231727 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fca545b-b330-4aa9-82d0-566ad0ac2c8c None admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:48.233158 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fca545b-b330-4aa9-82d0-566ad0ac2c8c None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:48.252808 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 98/197] 10.222.0.27 () {60 vars in 1132 bytes} [Sun Sep 11 00:38:48 2022] POST /identity/v3/users => generated 297 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:38:48.253859 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fca545b-b330-4aa9-82d0-566ad0ac2c8c None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:48.264447 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2220281-175d-44d7-a8e6-538f4b009e33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:48.265110 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2220281-175d-44d7-a8e6-538f4b009e33 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:48.265387 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2220281-175d-44d7-a8e6-538f4b009e33 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:48.265658 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2220281-175d-44d7-a8e6-538f4b009e33 None admin] PATH_INFO: `/v3/users/f4b18ce48c9e4f2196e1fa77aec45f6f` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:48.267048 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2220281-175d-44d7-a8e6-538f4b009e33 None admin] RBAC: Authorizing `identity:get_user(user_id=f4b18ce48c9e4f2196e1fa77aec45f6f)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:48.268733 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2220281-175d-44d7-a8e6-538f4b009e33 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:48.275193 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 100/198] 10.222.0.27 () {56 vars in 1177 bytes} [Sun Sep 11 00:38:48 2022] GET /identity/v3/users/f4b18ce48c9e4f2196e1fa77aec45f6f => generated 283 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:48.275193 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2220281-175d-44d7-a8e6-538f4b009e33 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:48.375465 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fca545b-b330-4aa9-82d0-566ad0ac2c8c None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:48.386591 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2220281-175d-44d7-a8e6-538f4b009e33 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:48.412412 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e4aced9-4542-4118-a650-8b03b58098ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:48.413135 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4aced9-4542-4118-a650-8b03b58098ef None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:48.413484 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4aced9-4542-4118-a650-8b03b58098ef None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:48.413829 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4aced9-4542-4118-a650-8b03b58098ef None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:48.414653 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e4aced9-4542-4118-a650-8b03b58098ef None admin] RBAC: Adding query filter params (role.id=7d9213c1187c4b5182bf8132cf23ea19, scope.project.id=1cc0eb0de66448aa8d924179a50e25e9, user.id=36edaa883d654921ac17e7745c0ff320) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:38:48.415064 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e4aced9-4542-4118-a650-8b03b58098ef None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:48.417102 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e4aced9-4542-4118-a650-8b03b58098ef None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:48.417294 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:48.424929 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 99/199] 10.222.0.27 () {56 vars in 1374 bytes} [Sun Sep 11 00:38:48 2022] GET /identity/v3/role_assignments?user.id=36edaa883d654921ac17e7745c0ff320&scope.project.id=1cc0eb0de66448aa8d924179a50e25e9&role.id=7d9213c1187c4b5182bf8132cf23ea19 => generated 608 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:48.426604 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e4aced9-4542-4118-a650-8b03b58098ef None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:48.428827 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:48.437503 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ccda5cd-727f-4cd1-91b7-9c3aa779c9be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:48.438308 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ccda5cd-727f-4cd1-91b7-9c3aa779c9be None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:48.438593 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ccda5cd-727f-4cd1-91b7-9c3aa779c9be None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:48.439491 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ccda5cd-727f-4cd1-91b7-9c3aa779c9be None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:48.538252 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ccda5cd-727f-4cd1-91b7-9c3aa779c9be None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:48.539876 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ccda5cd-727f-4cd1-91b7-9c3aa779c9be None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:48.578704 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e4aced9-4542-4118-a650-8b03b58098ef None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:48.631793 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb46949-6b2b-413c-b853-fd0888657930 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:48.631793 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb46949-6b2b-413c-b853-fd0888657930 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:48.632858 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb46949-6b2b-413c-b853-fd0888657930 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:48.632858 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb46949-6b2b-413c-b853-fd0888657930 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:48.686968 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 101/200] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:38:48 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 415 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 11 00:38:48.687535 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ccda5cd-727f-4cd1-91b7-9c3aa779c9be None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:48.722554 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:48.736471 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb46949-6b2b-413c-b853-fd0888657930 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:48.738109 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb46949-6b2b-413c-b853-fd0888657930 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:48.754455 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:48.805722 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ccda5cd-727f-4cd1-91b7-9c3aa779c9be None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:48.846938 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef16c28-1e41-4bb3-babc-775d9f6723da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:48.847756 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef16c28-1e41-4bb3-babc-775d9f6723da None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:48.848130 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef16c28-1e41-4bb3-babc-775d9f6723da None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:48.848497 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef16c28-1e41-4bb3-babc-775d9f6723da None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:48.863511 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 100/201] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:38:48 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 438 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 11 00:38:48.864856 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb46949-6b2b-413c-b853-fd0888657930 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:48.922314 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eef16c28-1e41-4bb3-babc-775d9f6723da None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:48.923910 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eef16c28-1e41-4bb3-babc-775d9f6723da None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:48.954980 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:49.004235 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb46949-6b2b-413c-b853-fd0888657930 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:49.006819 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:49.041905 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 102/202] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:38:48 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 355 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 11 00:38:49.043121 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef16c28-1e41-4bb3-babc-775d9f6723da None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:49.053490 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb941d0b-1178-49ac-989d-d1f28990abec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:49.054133 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb941d0b-1178-49ac-989d-d1f28990abec None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:49.054410 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb941d0b-1178-49ac-989d-d1f28990abec None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:49.054676 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb941d0b-1178-49ac-989d-d1f28990abec None admin] PATH_INFO: `/v3/projects/a3af44366c3a450994b868f0b5428c43` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:49.055957 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb941d0b-1178-49ac-989d-d1f28990abec None admin] RBAC: Authorizing `identity:get_project(project_id=a3af44366c3a450994b868f0b5428c43)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:49.057600 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb941d0b-1178-49ac-989d-d1f28990abec None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:49.060857 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 101/203] 10.222.0.27 () {56 vars in 1186 bytes} [Sun Sep 11 00:38:48 2022] GET /identity/v3/projects/a3af44366c3a450994b868f0b5428c43 => generated 302 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:49.060857 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb941d0b-1178-49ac-989d-d1f28990abec None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:49.172887 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef16c28-1e41-4bb3-babc-775d9f6723da None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:49.187418 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb941d0b-1178-49ac-989d-d1f28990abec None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:49.210243 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:49.224287 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e14019a-e61a-453d-8820-6711b694ec9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:49.225429 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e14019a-e61a-453d-8820-6711b694ec9b None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:49.225956 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e14019a-e61a-453d-8820-6711b694ec9b None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:49.226437 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e14019a-e61a-453d-8820-6711b694ec9b None admin] PATH_INFO: `/v3/projects/1cc0eb0de66448aa8d924179a50e25e9` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:49.227745 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da34cb7-4ec7-4b32-95a6-43af076941b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:49.228642 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e14019a-e61a-453d-8820-6711b694ec9b None admin] RBAC: Authorizing `identity:get_project(project_id=1cc0eb0de66448aa8d924179a50e25e9)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:49.229113 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da34cb7-4ec7-4b32-95a6-43af076941b3 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:49.229455 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da34cb7-4ec7-4b32-95a6-43af076941b3 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:49.229794 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da34cb7-4ec7-4b32-95a6-43af076941b3 None admin] PATH_INFO: `/v3/projects/a3af44366c3a450994b868f0b5428c43` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:49.231122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2da34cb7-4ec7-4b32-95a6-43af076941b3 None admin] RBAC: Authorizing `identity:get_project(project_id=a3af44366c3a450994b868f0b5428c43)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:49.231393 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e14019a-e61a-453d-8820-6711b694ec9b None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:49.232694 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2da34cb7-4ec7-4b32-95a6-43af076941b3 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:49.233495 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 103/204] 10.222.0.27 () {56 vars in 1186 bytes} [Sun Sep 11 00:38:49 2022] GET /identity/v3/projects/1cc0eb0de66448aa8d924179a50e25e9 => generated 298 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:49.234551 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e14019a-e61a-453d-8820-6711b694ec9b None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:49.234984 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 102/205] 10.222.0.27 () {56 vars in 1186 bytes} [Sun Sep 11 00:38:49 2022] GET /identity/v3/projects/a3af44366c3a450994b868f0b5428c43 => generated 302 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:49.240425 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da34cb7-4ec7-4b32-95a6-43af076941b3 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:49.293904 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:49.365291 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da34cb7-4ec7-4b32-95a6-43af076941b3 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:49.379711 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e14019a-e61a-453d-8820-6711b694ec9b None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:49.403928 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-388a284a-e48e-4ad7-8031-21982070c369 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:49.404700 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-388a284a-e48e-4ad7-8031-21982070c369 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:49.405049 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-388a284a-e48e-4ad7-8031-21982070c369 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:49.405390 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-388a284a-e48e-4ad7-8031-21982070c369 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:49.406203 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-388a284a-e48e-4ad7-8031-21982070c369 None admin] RBAC: Adding query filter params (role.id=7d473d5a7be94698b3b0ed9a2f499a56, scope.project.id=1cc0eb0de66448aa8d924179a50e25e9, user.id=67cdfca3c1f54228aab73b276b6d34bb) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:38:49.406574 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-388a284a-e48e-4ad7-8031-21982070c369 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:49.408129 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-388a284a-e48e-4ad7-8031-21982070c369 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:49.416271 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 103/206] 10.222.0.27 () {56 vars in 1374 bytes} [Sun Sep 11 00:38:49 2022] GET /identity/v3/role_assignments?user.id=67cdfca3c1f54228aab73b276b6d34bb&scope.project.id=1cc0eb0de66448aa8d924179a50e25e9&role.id=7d473d5a7be94698b3b0ed9a2f499a56 => generated 608 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:49.417263 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-388a284a-e48e-4ad7-8031-21982070c369 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:49.426924 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-994642d6-308b-4833-b6a3-cd41099b1861 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:49.428706 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994642d6-308b-4833-b6a3-cd41099b1861 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:49.429186 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994642d6-308b-4833-b6a3-cd41099b1861 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:49.429638 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994642d6-308b-4833-b6a3-cd41099b1861 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:49.431895 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-994642d6-308b-4833-b6a3-cd41099b1861 None admin] RBAC: Adding query filter params (role.id=7d9213c1187c4b5182bf8132cf23ea19, scope.project.id=a3af44366c3a450994b868f0b5428c43, user.id=f4b18ce48c9e4f2196e1fa77aec45f6f) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:38:49.432508 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-994642d6-308b-4833-b6a3-cd41099b1861 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:49.435133 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-994642d6-308b-4833-b6a3-cd41099b1861 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:49.444529 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 104/207] 10.222.0.27 () {56 vars in 1374 bytes} [Sun Sep 11 00:38:49 2022] GET /identity/v3/role_assignments?user.id=f4b18ce48c9e4f2196e1fa77aec45f6f&scope.project.id=a3af44366c3a450994b868f0b5428c43&role.id=7d9213c1187c4b5182bf8132cf23ea19 => generated 259 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:49.560878 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-388a284a-e48e-4ad7-8031-21982070c369 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:49.562888 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:49.617021 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67acdefe-2da4-4ce1-8a42-86654413ac7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:49.617640 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67acdefe-2da4-4ce1-8a42-86654413ac7a None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:49.617913 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67acdefe-2da4-4ce1-8a42-86654413ac7a None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:49.618177 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67acdefe-2da4-4ce1-8a42-86654413ac7a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:49.619593 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67acdefe-2da4-4ce1-8a42-86654413ac7a None admin] RBAC: Adding query filter params (role.id=518eab0f758241369edcdd997b98ad12, scope.project.id=a3af44366c3a450994b868f0b5428c43, user.id=f4b18ce48c9e4f2196e1fa77aec45f6f) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:38:49.620149 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67acdefe-2da4-4ce1-8a42-86654413ac7a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:49.621737 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67acdefe-2da4-4ce1-8a42-86654413ac7a None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:49.631558 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 104/208] 10.222.0.27 () {56 vars in 1374 bytes} [Sun Sep 11 00:38:49 2022] GET /identity/v3/role_assignments?user.id=f4b18ce48c9e4f2196e1fa77aec45f6f&scope.project.id=a3af44366c3a450994b868f0b5428c43&role.id=518eab0f758241369edcdd997b98ad12 => generated 259 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:50.745540 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8fbbd3-b8f1-44a3-a7f4-5171f923c0d6 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:50.746094 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8fbbd3-b8f1-44a3-a7f4-5171f923c0d6 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:50.746094 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8fbbd3-b8f1-44a3-a7f4-5171f923c0d6 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:50.747584 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 105/209] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:38:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:38:50.752961 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5cd1be-346c-4214-bb59-7e4416f318a2 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:50.753748 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5cd1be-346c-4214-bb59-7e4416f318a2 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:50.754236 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5cd1be-346c-4214-bb59-7e4416f318a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:50.774114 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-af5cd1be-346c-4214-bb59-7e4416f318a2 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:38:50.776334 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-af5cd1be-346c-4214-bb59-7e4416f318a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:38:50.849439 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 105/210] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:38:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 11 00:38:50.892878 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13d45429-7ae5-4fb6-bcf0-24be20857d5f None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:50.893204 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13d45429-7ae5-4fb6-bcf0-24be20857d5f None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:50.893436 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13d45429-7ae5-4fb6-bcf0-24be20857d5f None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:50.894778 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 106/211] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:38:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:38:50.900774 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4efacbdd-e00f-4115-adb9-623264312a56 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:50.901083 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4efacbdd-e00f-4115-adb9-623264312a56 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:50.901358 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4efacbdd-e00f-4115-adb9-623264312a56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:50.919722 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-4efacbdd-e00f-4115-adb9-623264312a56 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:38:50.920901 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-4efacbdd-e00f-4115-adb9-623264312a56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:38:50.988398 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 106/212] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:38:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:38:51.133208 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13d45429-7ae5-4fb6-bcf0-24be20857d5f None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:51.195616 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:51.242821 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13d45429-7ae5-4fb6-bcf0-24be20857d5f None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:51.268973 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4efacbdd-e00f-4115-adb9-623264312a56 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:51.279417 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f7e85a3-c79f-48dc-bc4b-195ad477a7bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:51.280053 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f7e85a3-c79f-48dc-bc4b-195ad477a7bb None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:51.280328 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f7e85a3-c79f-48dc-bc4b-195ad477a7bb None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:51.280651 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f7e85a3-c79f-48dc-bc4b-195ad477a7bb None admin] PATH_INFO: `/v3/roles/member` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:51.292828 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f7e85a3-c79f-48dc-bc4b-195ad477a7bb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:38:51.298356 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:51.299220 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-0f7e85a3-c79f-48dc-bc4b-195ad477a7bb None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 11 00:38:51.299220 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:38:51.299220 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:38:51.299220 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:38:51.299220 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:38:51.299220 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:38:51.299220 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:38:51.299220 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:38:51.299220 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:38:51.299220 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:38:51.299220 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:38:51.299220 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:38:51.299220 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:38:51.299220 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:38:51.299220 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:38:51.299220 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:38:51.300089 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:38:51.300089 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:38:51.300089 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:38:51.300089 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:38:51.300089 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:38:51.300089 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:38:51.300089 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:38:51.300089 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:38:51.300089 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:38:51.300089 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:38:51.300089 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:38:51.300089 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:38:51.300089 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:38:51.300089 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:38:51.300089 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:38:51.300089 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:38:51.300880 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:38:51.300880 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:38:51.300880 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:38:51.300880 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:38:51.300880 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:38:51.300880 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:38:51.300880 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:38:51.300880 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:38:51.300880 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:38:51.300880 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:38:51.300880 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 11 00:38:51.300880 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:38:51.300880 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 107/213] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:38:51 2022] GET /identity/v3/roles/member => generated 84 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:38:51.308646 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f7e85a3-c79f-48dc-bc4b-195ad477a7bb None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:51.394034 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4efacbdd-e00f-4115-adb9-623264312a56 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:51.439227 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f7e85a3-c79f-48dc-bc4b-195ad477a7bb None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:51.440469 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ffa82e3-fa8f-429f-837b-72915de9df30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:51.442485 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ffa82e3-fa8f-429f-837b-72915de9df30 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:51.443040 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ffa82e3-fa8f-429f-837b-72915de9df30 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:51.443503 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ffa82e3-fa8f-429f-837b-72915de9df30 None admin] PATH_INFO: `/v3/domains/default` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:51.445022 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ffa82e3-fa8f-429f-837b-72915de9df30 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:51.447518 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ffa82e3-fa8f-429f-837b-72915de9df30 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:51.454939 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 107/214] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:38:51 2022] GET /identity/v3/domains/default => generated 199 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:51.455777 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ffa82e3-fa8f-429f-837b-72915de9df30 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:51.464112 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:51.486156 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a1eefb2-9809-4728-8435-7f0f8b666d4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:51.486760 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a1eefb2-9809-4728-8435-7f0f8b666d4d None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:51.487055 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a1eefb2-9809-4728-8435-7f0f8b666d4d None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:51.487322 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a1eefb2-9809-4728-8435-7f0f8b666d4d None admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:51.487984 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a1eefb2-9809-4728-8435-7f0f8b666d4d None admin] RBAC: Adding query filter params (name=member) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:38:51.488283 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a1eefb2-9809-4728-8435-7f0f8b666d4d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:51.489663 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a1eefb2-9809-4728-8435-7f0f8b666d4d None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:51.499041 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 108/215] 10.222.0.27 () {56 vars in 1101 bytes} [Sun Sep 11 00:38:51 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:38:51.505391 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a1eefb2-9809-4728-8435-7f0f8b666d4d None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:51.587124 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:51.597829 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ffa82e3-fa8f-429f-837b-72915de9df30 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:51.636658 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a1eefb2-9809-4728-8435-7f0f8b666d4d None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:51.637211 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e5e648-7608-4101-83a1-7869ccee44f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:51.637923 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e5e648-7608-4101-83a1-7869ccee44f7 None admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:51.638295 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e5e648-7608-4101-83a1-7869ccee44f7 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:51.638658 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e5e648-7608-4101-83a1-7869ccee44f7 None admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:51.639425 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4e5e648-7608-4101-83a1-7869ccee44f7 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:51.640956 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4e5e648-7608-4101-83a1-7869ccee44f7 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:51.675111 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 108/216] 10.222.0.27 () {60 vars in 1132 bytes} [Sun Sep 11 00:38:51 2022] POST /identity/v3/users => generated 297 bytes in 213 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:38:51.684611 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83957669-b5ef-464a-8288-b7c05c29bc4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:51.685350 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83957669-b5ef-464a-8288-b7c05c29bc4d None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:51.685625 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83957669-b5ef-464a-8288-b7c05c29bc4d None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:51.685894 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83957669-b5ef-464a-8288-b7c05c29bc4d None admin] PATH_INFO: `/v3/users/e27b22a7b6a643b492a02622cf3fde8a` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:51.706059 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83957669-b5ef-464a-8288-b7c05c29bc4d None admin] RBAC: Authorizing `identity:get_user(user_id=e27b22a7b6a643b492a02622cf3fde8a)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:51.707743 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83957669-b5ef-464a-8288-b7c05c29bc4d None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:51.709631 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 109/217] 10.222.0.27 () {56 vars in 1177 bytes} [Sun Sep 11 00:38:51 2022] GET /identity/v3/users/e27b22a7b6a643b492a02622cf3fde8a => generated 297 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:51.716603 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e5e648-7608-4101-83a1-7869ccee44f7 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:51.821103 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a44f56c-e1d5-42ae-9040-c418606d9855 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:51.821669 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a44f56c-e1d5-42ae-9040-c418606d9855 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:51.821669 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a44f56c-e1d5-42ae-9040-c418606d9855 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:51.823074 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 110/218] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:38:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 11 00:38:51.828644 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5fb82c-9b04-4ace-8a7d-102044c36e97 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:51.828935 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5fb82c-9b04-4ace-8a7d-102044c36e97 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:51.829187 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5fb82c-9b04-4ace-8a7d-102044c36e97 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:51.841309 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e5e648-7608-4101-83a1-7869ccee44f7 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:51.849078 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-3a5fb82c-9b04-4ace-8a7d-102044c36e97 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:38:51.850473 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-3a5fb82c-9b04-4ace-8a7d-102044c36e97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:38:51.893246 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3acc1c6-95ed-463e-ac8a-e8a02bd3bff3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:51.894111 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3acc1c6-95ed-463e-ac8a-e8a02bd3bff3 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:51.894111 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3acc1c6-95ed-463e-ac8a-e8a02bd3bff3 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:51.894399 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3acc1c6-95ed-463e-ac8a-e8a02bd3bff3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:51.894693 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:51.922332 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:51.952061 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 111/219] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:38:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 11 00:38:51.979112 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b81b97-a45a-4ae5-9d3a-b62faa3cbbb4 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:51.979718 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b81b97-a45a-4ae5-9d3a-b62faa3cbbb4 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:51.980224 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b81b97-a45a-4ae5-9d3a-b62faa3cbbb4 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:51.981759 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 112/220] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:38:51 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 11 00:38:51.988531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbca652f-df86-4790-899d-f3a6c6f34f5a None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:51.988809 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbca652f-df86-4790-899d-f3a6c6f34f5a None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:51.989076 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbca652f-df86-4790-899d-f3a6c6f34f5a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:51.997447 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3acc1c6-95ed-463e-ac8a-e8a02bd3bff3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:51.999584 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3acc1c6-95ed-463e-ac8a-e8a02bd3bff3 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:52.013559 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-bbca652f-df86-4790-899d-f3a6c6f34f5a None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:38:52.014749 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-bbca652f-df86-4790-899d-f3a6c6f34f5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:38:52.105741 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 113/221] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:38:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:38:52.141279 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 109/222] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:38:51 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 421 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 11 00:38:52.142780 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbca652f-df86-4790-899d-f3a6c6f34f5a None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:52.249026 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:52.249224 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3acc1c6-95ed-463e-ac8a-e8a02bd3bff3 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:52.253347 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbca652f-df86-4790-899d-f3a6c6f34f5a None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:52.295356 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c90973fd-2d2a-4e43-a30d-3ed743c33169 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:52.296057 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90973fd-2d2a-4e43-a30d-3ed743c33169 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:52.296332 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90973fd-2d2a-4e43-a30d-3ed743c33169 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:52.296599 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90973fd-2d2a-4e43-a30d-3ed743c33169 None admin] PATH_INFO: `/v3/projects/a3af44366c3a450994b868f0b5428c43` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:52.297783 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c90973fd-2d2a-4e43-a30d-3ed743c33169 None admin] RBAC: Authorizing `identity:get_project(project_id=a3af44366c3a450994b868f0b5428c43)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:52.299291 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c90973fd-2d2a-4e43-a30d-3ed743c33169 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:52.304335 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 114/223] 10.222.0.27 () {56 vars in 1186 bytes} [Sun Sep 11 00:38:52 2022] GET /identity/v3/projects/a3af44366c3a450994b868f0b5428c43 => generated 302 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:52.307797 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c90973fd-2d2a-4e43-a30d-3ed743c33169 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:52.327203 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:52.376244 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3acc1c6-95ed-463e-ac8a-e8a02bd3bff3 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:52.435805 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1afe7b1c-a03f-45b7-8e69-50fd9a6e21da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:52.436444 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1afe7b1c-a03f-45b7-8e69-50fd9a6e21da None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:52.436781 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1afe7b1c-a03f-45b7-8e69-50fd9a6e21da None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:52.437053 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1afe7b1c-a03f-45b7-8e69-50fd9a6e21da None admin] PATH_INFO: `/v3/roles/admin` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:52.449451 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1afe7b1c-a03f-45b7-8e69-50fd9a6e21da None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:38:52.454943 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-1afe7b1c-a03f-45b7-8e69-50fd9a6e21da None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:38:52.454943 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:38:52.454943 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:38:52.454943 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:38:52.454943 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:38:52.454943 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:38:52.454943 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:38:52.454943 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:38:52.454943 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:38:52.454943 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:38:52.454943 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:38:52.454943 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:38:52.454943 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:38:52.454943 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:38:52.454943 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:38:52.454943 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:38:52.455913 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:38:52.455913 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:38:52.455913 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:38:52.455913 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:38:52.455913 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:38:52.455913 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:38:52.455913 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:38:52.455913 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:38:52.455913 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:38:52.455913 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:38:52.455913 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:38:52.455913 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:38:52.455913 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:38:52.455913 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:38:52.455913 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:38:52.455913 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:38:52.456758 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:38:52.456758 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:38:52.456758 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:38:52.456758 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:38:52.456758 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:38:52.456758 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:38:52.456758 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:38:52.456758 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:38:52.456758 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:38:52.456758 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:38:52.456758 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:38:52.456758 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:38:52.456758 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 110/224] 10.222.0.27 () {56 vars in 1096 bytes} [Sun Sep 11 00:38:52 2022] GET /identity/v3/roles/admin => generated 83 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:38:52.464546 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1afe7b1c-a03f-45b7-8e69-50fd9a6e21da None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:52.474574 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c90973fd-2d2a-4e43-a30d-3ed743c33169 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:52.511337 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5089888f-293f-48c2-848c-49764ff2149f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:52.511865 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5089888f-293f-48c2-848c-49764ff2149f None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:52.512202 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5089888f-293f-48c2-848c-49764ff2149f None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:52.512474 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5089888f-293f-48c2-848c-49764ff2149f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:52.513256 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5089888f-293f-48c2-848c-49764ff2149f None admin] RBAC: Adding query filter params (role.id=2e64fc597b6447df99a7992d411d1a34, scope.project.id=a3af44366c3a450994b868f0b5428c43, user.id=e27b22a7b6a643b492a02622cf3fde8a) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:38:52.513566 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5089888f-293f-48c2-848c-49764ff2149f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:52.515029 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5089888f-293f-48c2-848c-49764ff2149f None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:52.527048 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 115/225] 10.222.0.27 () {56 vars in 1374 bytes} [Sun Sep 11 00:38:52 2022] GET /identity/v3/role_assignments?user.id=e27b22a7b6a643b492a02622cf3fde8a&scope.project.id=a3af44366c3a450994b868f0b5428c43&role.id=2e64fc597b6447df99a7992d411d1a34 => generated 259 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:52.528105 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5089888f-293f-48c2-848c-49764ff2149f None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:52.586652 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:52.588162 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1afe7b1c-a03f-45b7-8e69-50fd9a6e21da None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:52.604068 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:52.634723 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d104fc-ab40-405b-b91f-df6b3506860d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:52.635789 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d104fc-ab40-405b-b91f-df6b3506860d None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:52.636071 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d104fc-ab40-405b-b91f-df6b3506860d None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:52.636343 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d104fc-ab40-405b-b91f-df6b3506860d None admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:52.637035 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6d104fc-ab40-405b-b91f-df6b3506860d None admin] RBAC: Adding query filter params (name=admin) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:38:52.637639 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6d104fc-ab40-405b-b91f-df6b3506860d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:52.639415 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6d104fc-ab40-405b-b91f-df6b3506860d None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:52.649675 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 111/226] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:38:52 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:52.656452 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d104fc-ab40-405b-b91f-df6b3506860d None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:52.740300 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5089888f-293f-48c2-848c-49764ff2149f None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:52.813832 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52628830-666f-4f2c-b7be-99bb94779071 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:52.814431 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52628830-666f-4f2c-b7be-99bb94779071 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:52.814717 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52628830-666f-4f2c-b7be-99bb94779071 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:52.815007 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52628830-666f-4f2c-b7be-99bb94779071 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:52.834839 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52628830-666f-4f2c-b7be-99bb94779071 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:38:52.838230 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d104fc-ab40-405b-b91f-df6b3506860d None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:52.840614 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-52628830-666f-4f2c-b7be-99bb94779071 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 11 00:38:52.840614 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:38:52.840614 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:38:52.840614 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:38:52.840614 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:38:52.840614 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:38:52.840614 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:38:52.840614 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:38:52.840614 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:38:52.840614 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:38:52.840614 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:38:52.840614 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:38:52.840614 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:38:52.840614 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:38:52.840614 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:38:52.840614 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:38:52.841496 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:38:52.841496 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:38:52.841496 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:38:52.841496 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:38:52.841496 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:38:52.841496 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:38:52.841496 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:38:52.841496 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:38:52.841496 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:38:52.841496 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:38:52.841496 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:38:52.841496 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:38:52.841496 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:38:52.841496 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:38:52.841496 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:38:52.841496 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:38:52.842305 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:38:52.842305 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:38:52.842305 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:38:52.842305 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:38:52.842305 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:38:52.842305 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:38:52.842305 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:38:52.842305 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:38:52.842305 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:38:52.842305 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:38:52.842305 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 11 00:38:52.842305 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:38:52.842305 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 116/227] 10.222.0.27 () {56 vars in 1114 bytes} [Sun Sep 11 00:38:52 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 314 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:38:52.850060 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52628830-666f-4f2c-b7be-99bb94779071 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:52.881686 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c255da65-1622-47e7-957c-1fb7e5c49884 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:52.883202 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c255da65-1622-47e7-957c-1fb7e5c49884 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:52.883680 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c255da65-1622-47e7-957c-1fb7e5c49884 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:52.884710 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c255da65-1622-47e7-957c-1fb7e5c49884 None admin] PATH_INFO: `/v3/users/f4b18ce48c9e4f2196e1fa77aec45f6f` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:52.886069 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c255da65-1622-47e7-957c-1fb7e5c49884 None admin] RBAC: Authorizing `identity:get_user(user_id=f4b18ce48c9e4f2196e1fa77aec45f6f)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:52.888075 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c255da65-1622-47e7-957c-1fb7e5c49884 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:52.890229 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 112/228] 10.222.0.27 () {56 vars in 1177 bytes} [Sun Sep 11 00:38:52 2022] GET /identity/v3/users/f4b18ce48c9e4f2196e1fa77aec45f6f => generated 283 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:52.895996 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c255da65-1622-47e7-957c-1fb7e5c49884 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:52.945338 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:53.006977 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c255da65-1622-47e7-957c-1fb7e5c49884 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:53.021319 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:53.045369 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52628830-666f-4f2c-b7be-99bb94779071 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:53.046254 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f58714e-6498-4398-b115-addf3ded5f2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:53.047156 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f58714e-6498-4398-b115-addf3ded5f2b None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:53.047628 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f58714e-6498-4398-b115-addf3ded5f2b None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:53.048090 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f58714e-6498-4398-b115-addf3ded5f2b None admin] PATH_INFO: `/v3/projects/a3af44366c3a450994b868f0b5428c43` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:53.049420 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f58714e-6498-4398-b115-addf3ded5f2b None admin] RBAC: Authorizing `identity:get_project(project_id=a3af44366c3a450994b868f0b5428c43)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:53.051717 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f58714e-6498-4398-b115-addf3ded5f2b None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:53.054681 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 113/229] 10.222.0.27 () {56 vars in 1186 bytes} [Sun Sep 11 00:38:52 2022] GET /identity/v3/projects/a3af44366c3a450994b868f0b5428c43 => generated 302 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:53.068516 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f58714e-6498-4398-b115-addf3ded5f2b None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:53.116677 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dec1795-857f-4c69-bfe7-e176790a1eac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:53.117766 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dec1795-857f-4c69-bfe7-e176790a1eac None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:53.118236 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dec1795-857f-4c69-bfe7-e176790a1eac None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:53.119425 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dec1795-857f-4c69-bfe7-e176790a1eac None admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:53.120568 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dec1795-857f-4c69-bfe7-e176790a1eac None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:38:53.121068 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dec1795-857f-4c69-bfe7-e176790a1eac None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:53.123311 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dec1795-857f-4c69-bfe7-e176790a1eac None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:53.143419 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 117/230] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:38:52 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 294 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:53.149124 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dec1795-857f-4c69-bfe7-e176790a1eac None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:53.254505 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f58714e-6498-4398-b115-addf3ded5f2b None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:53.273737 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:53.293898 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e2438b9-020a-4ff2-b20b-04b66c9db7b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:53.294562 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2438b9-020a-4ff2-b20b-04b66c9db7b8 None admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:53.294861 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2438b9-020a-4ff2-b20b-04b66c9db7b8 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:53.295138 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2438b9-020a-4ff2-b20b-04b66c9db7b8 None admin] PATH_INFO: `/v3/projects/a3af44366c3a450994b868f0b5428c43/users/f4b18ce48c9e4f2196e1fa77aec45f6f/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:53.297374 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e2438b9-020a-4ff2-b20b-04b66c9db7b8 None admin] RBAC: Authorizing `identity:create_grant(project_id=a3af44366c3a450994b868f0b5428c43, user_id=f4b18ce48c9e4f2196e1fa77aec45f6f, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:53.299256 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e2438b9-020a-4ff2-b20b-04b66c9db7b8 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:53.314001 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 114/231] 10.222.0.27 () {58 vars in 1439 bytes} [Sun Sep 11 00:38:53 2022] PUT /identity/v3/projects/a3af44366c3a450994b868f0b5428c43/users/f4b18ce48c9e4f2196e1fa77aec45f6f/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 246 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:38:53.395264 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dec1795-857f-4c69-bfe7-e176790a1eac None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:53.450745 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc834f12-f25e-4ef3-b57a-776f7ead697e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:53.451443 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc834f12-f25e-4ef3-b57a-776f7ead697e None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:53.451728 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc834f12-f25e-4ef3-b57a-776f7ead697e None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:53.452006 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc834f12-f25e-4ef3-b57a-776f7ead697e None admin] PATH_INFO: `/v3/users/f4b18ce48c9e4f2196e1fa77aec45f6f` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:53.453319 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc834f12-f25e-4ef3-b57a-776f7ead697e None admin] RBAC: Authorizing `identity:get_user(user_id=f4b18ce48c9e4f2196e1fa77aec45f6f)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:53.454825 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc834f12-f25e-4ef3-b57a-776f7ead697e None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:53.456910 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 118/232] 10.222.0.27 () {56 vars in 1177 bytes} [Sun Sep 11 00:38:53 2022] GET /identity/v3/users/f4b18ce48c9e4f2196e1fa77aec45f6f => generated 283 bytes in 308 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:53.463648 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e2438b9-020a-4ff2-b20b-04b66c9db7b8 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:53.607931 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e2438b9-020a-4ff2-b20b-04b66c9db7b8 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:53.644721 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e2d066b-e64c-4c68-8b84-2bd4ea7d9353 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:53.645726 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2d066b-e64c-4c68-8b84-2bd4ea7d9353 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:53.646187 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2d066b-e64c-4c68-8b84-2bd4ea7d9353 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:53.646653 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2d066b-e64c-4c68-8b84-2bd4ea7d9353 None admin] PATH_INFO: `/v3/projects/a3af44366c3a450994b868f0b5428c43` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:53.648009 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e2d066b-e64c-4c68-8b84-2bd4ea7d9353 None admin] RBAC: Authorizing `identity:get_project(project_id=a3af44366c3a450994b868f0b5428c43)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:53.649748 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e2d066b-e64c-4c68-8b84-2bd4ea7d9353 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:53.651798 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 115/233] 10.222.0.27 () {56 vars in 1186 bytes} [Sun Sep 11 00:38:53 2022] GET /identity/v3/projects/a3af44366c3a450994b868f0b5428c43 => generated 302 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:53.657698 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc834f12-f25e-4ef3-b57a-776f7ead697e None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:53.681809 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:53.765844 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc834f12-f25e-4ef3-b57a-776f7ead697e None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:53.803689 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cddf485-34e7-45ba-bbb4-1a7329aa0375 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:53.804417 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cddf485-34e7-45ba-bbb4-1a7329aa0375 None admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:53.804703 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cddf485-34e7-45ba-bbb4-1a7329aa0375 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:53.805025 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cddf485-34e7-45ba-bbb4-1a7329aa0375 None admin] PATH_INFO: `/v3/projects/a3af44366c3a450994b868f0b5428c43/users/f4b18ce48c9e4f2196e1fa77aec45f6f/roles/518eab0f758241369edcdd997b98ad12` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:53.808039 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cddf485-34e7-45ba-bbb4-1a7329aa0375 None admin] RBAC: Authorizing `identity:create_grant(project_id=a3af44366c3a450994b868f0b5428c43, user_id=f4b18ce48c9e4f2196e1fa77aec45f6f, role_id=518eab0f758241369edcdd997b98ad12)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:53.809502 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cddf485-34e7-45ba-bbb4-1a7329aa0375 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:53.832686 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 119/234] 10.222.0.27 () {58 vars in 1439 bytes} [Sun Sep 11 00:38:53 2022] PUT /identity/v3/projects/a3af44366c3a450994b868f0b5428c43/users/f4b18ce48c9e4f2196e1fa77aec45f6f/roles/518eab0f758241369edcdd997b98ad12 => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:38:54.330136 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453fb280-9761-4b95-bc8b-ce8adc78e35d None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:54.330652 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453fb280-9761-4b95-bc8b-ce8adc78e35d None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:54.330652 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453fb280-9761-4b95-bc8b-ce8adc78e35d None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:54.332048 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 116/235] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:38:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:38:54.346700 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2f2339-391a-42d9-b83b-ea0abe67a238 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:54.347239 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2f2339-391a-42d9-b83b-ea0abe67a238 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:54.347315 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2f2339-391a-42d9-b83b-ea0abe67a238 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:54.375925 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-6e2f2339-391a-42d9-b83b-ea0abe67a238 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:38:54.377037 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-6e2f2339-391a-42d9-b83b-ea0abe67a238 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:38:54.461635 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 120/236] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:38:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:38:54.631682 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90c89e4d-7671-482c-b269-3cdff5011ff0 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:54.632030 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90c89e4d-7671-482c-b269-3cdff5011ff0 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:54.632134 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90c89e4d-7671-482c-b269-3cdff5011ff0 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:54.633480 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 117/237] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:38:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:38:54.639400 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9308d9f-b17c-4eed-b543-1e51bc159559 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:54.639678 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9308d9f-b17c-4eed-b543-1e51bc159559 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:54.639945 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9308d9f-b17c-4eed-b543-1e51bc159559 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:54.659128 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-c9308d9f-b17c-4eed-b543-1e51bc159559 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:38:54.660568 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-c9308d9f-b17c-4eed-b543-1e51bc159559 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:38:54.663487 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:54.749029 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 121/238] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:38:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:38:54.892491 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90c89e4d-7671-482c-b269-3cdff5011ff0 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:54.930648 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:55.001512 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90c89e4d-7671-482c-b269-3cdff5011ff0 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:55.047060 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7506741d-9028-44d4-b22d-1c7ea50641dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:55.047700 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7506741d-9028-44d4-b22d-1c7ea50641dd None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:55.047976 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7506741d-9028-44d4-b22d-1c7ea50641dd None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:55.048249 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7506741d-9028-44d4-b22d-1c7ea50641dd None admin] PATH_INFO: `/v3/roles/reader` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:55.066759 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7506741d-9028-44d4-b22d-1c7ea50641dd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:38:55.072701 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-7506741d-9028-44d4-b22d-1c7ea50641dd None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 11 00:38:55.072701 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:38:55.072701 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:38:55.072701 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:38:55.072701 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:38:55.072701 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:38:55.072701 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:38:55.072701 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:38:55.072701 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:38:55.072701 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:38:55.072701 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:38:55.072701 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:38:55.072701 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:38:55.072701 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:38:55.072701 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:38:55.072701 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:38:55.073655 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:38:55.073655 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:38:55.073655 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:38:55.073655 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:38:55.073655 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:38:55.073655 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:38:55.073655 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:38:55.073655 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:38:55.073655 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:38:55.073655 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:38:55.073655 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:38:55.073655 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:38:55.073655 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:38:55.073655 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:38:55.073655 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:38:55.073655 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:38:55.074453 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:38:55.074453 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:38:55.074453 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:38:55.074453 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:38:55.074453 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:38:55.074453 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:38:55.074453 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:38:55.074453 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:38:55.074453 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:38:55.074453 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:38:55.074453 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 11 00:38:55.074453 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:38:55.074453 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 118/239] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:38:54 2022] GET /identity/v3/roles/reader => generated 84 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:38:55.083943 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9308d9f-b17c-4eed-b543-1e51bc159559 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:55.126010 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7506741d-9028-44d4-b22d-1c7ea50641dd None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:55.273572 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9308d9f-b17c-4eed-b543-1e51bc159559 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:55.308288 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:55.318657 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7506741d-9028-44d4-b22d-1c7ea50641dd None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:55.329859 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e34936fb-5c93-4473-bd69-19c63fe06eeb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:55.330750 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e34936fb-5c93-4473-bd69-19c63fe06eeb None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:55.330881 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e34936fb-5c93-4473-bd69-19c63fe06eeb None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:55.331149 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e34936fb-5c93-4473-bd69-19c63fe06eeb None admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:55.331830 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e34936fb-5c93-4473-bd69-19c63fe06eeb None admin] RBAC: Adding query filter params (name=reader) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:38:55.332134 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e34936fb-5c93-4473-bd69-19c63fe06eeb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:55.333631 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e34936fb-5c93-4473-bd69-19c63fe06eeb None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:55.346836 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 122/240] 10.222.0.27 () {56 vars in 1101 bytes} [Sun Sep 11 00:38:55 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:55.346836 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd0fa1a-88f4-4843-bbde-cddda86554ae None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:55.346836 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd0fa1a-88f4-4843-bbde-cddda86554ae None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:55.346836 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd0fa1a-88f4-4843-bbde-cddda86554ae None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:55.346836 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 123/241] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:38:55 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:38:55.352542 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd0fa1a-88f4-4843-bbde-cddda86554ae None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:55.355468 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:55.377104 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4642a033-677a-431b-9db9-c47679227da8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:55.377771 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4642a033-677a-431b-9db9-c47679227da8 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:55.378027 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4642a033-677a-431b-9db9-c47679227da8 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:55.378303 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4642a033-677a-431b-9db9-c47679227da8 None admin] PATH_INFO: `/v3/domains/default` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:55.379525 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4642a033-677a-431b-9db9-c47679227da8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:55.381590 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4642a033-677a-431b-9db9-c47679227da8 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:55.386854 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 119/242] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:38:55 2022] GET /identity/v3/domains/default => generated 199 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:38:55.388270 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2e426b-5ef7-41ef-aa9b-f1ca854ccca6 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:55.388836 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2e426b-5ef7-41ef-aa9b-f1ca854ccca6 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:55.389511 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2e426b-5ef7-41ef-aa9b-f1ca854ccca6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:55.408385 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-ab2e426b-5ef7-41ef-aa9b-f1ca854ccca6 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:38:55.409575 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-ab2e426b-5ef7-41ef-aa9b-f1ca854ccca6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:38:55.492036 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 120/243] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:38:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:38:55.493005 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab2e426b-5ef7-41ef-aa9b-f1ca854ccca6 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:55.503578 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd0fa1a-88f4-4843-bbde-cddda86554ae None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:55.546556 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-685008fe-1fa7-4d8a-b723-1e0c089d0dc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:55.547154 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685008fe-1fa7-4d8a-b723-1e0c089d0dc5 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:55.547477 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685008fe-1fa7-4d8a-b723-1e0c089d0dc5 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:55.547790 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685008fe-1fa7-4d8a-b723-1e0c089d0dc5 None admin] PATH_INFO: `/v3/users/0bb1412f97544e96bcaa961924138d03` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:55.579123 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-685008fe-1fa7-4d8a-b723-1e0c089d0dc5 None admin] RBAC: Authorizing `identity:get_user(user_id=0bb1412f97544e96bcaa961924138d03)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:55.581137 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-685008fe-1fa7-4d8a-b723-1e0c089d0dc5 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:55.583932 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 124/244] 10.222.0.27 () {56 vars in 1177 bytes} [Sun Sep 11 00:38:55 2022] GET /identity/v3/users/0bb1412f97544e96bcaa961924138d03 => generated 297 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:55.590664 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-685008fe-1fa7-4d8a-b723-1e0c089d0dc5 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:55.669344 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:55.678397 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab2e426b-5ef7-41ef-aa9b-f1ca854ccca6 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:55.704171 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:55.740229 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c28d26c4-546e-4ef8-a726-52440eac4533 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:55.740848 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28d26c4-546e-4ef8-a726-52440eac4533 None admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:55.741121 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28d26c4-546e-4ef8-a726-52440eac4533 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:55.741378 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28d26c4-546e-4ef8-a726-52440eac4533 None admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:55.742035 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c28d26c4-546e-4ef8-a726-52440eac4533 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:55.744479 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c28d26c4-546e-4ef8-a726-52440eac4533 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:55.750487 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-685008fe-1fa7-4d8a-b723-1e0c089d0dc5 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:55.785498 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 121/245] 10.222.0.27 () {60 vars in 1132 bytes} [Sun Sep 11 00:38:55 2022] POST /identity/v3/users => generated 293 bytes in 293 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:38:55.787637 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd751b2-69fd-4fd4-be1c-6bc335282592 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:55.788413 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd751b2-69fd-4fd4-be1c-6bc335282592 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:55.788834 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd751b2-69fd-4fd4-be1c-6bc335282592 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:55.789273 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd751b2-69fd-4fd4-be1c-6bc335282592 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:55.863295 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c28d26c4-546e-4ef8-a726-52440eac4533 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:55.870625 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cd751b2-69fd-4fd4-be1c-6bc335282592 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:55.872364 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cd751b2-69fd-4fd4-be1c-6bc335282592 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:55.936246 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:55.986035 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 125/246] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:38:55 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 396 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 11 00:38:55.992278 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd751b2-69fd-4fd4-be1c-6bc335282592 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:56.025248 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c28d26c4-546e-4ef8-a726-52440eac4533 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:56.095555 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc6696af-2cfc-4330-9af4-d2630c93db00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:56.096292 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6696af-2cfc-4330-9af4-d2630c93db00 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:56.096572 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6696af-2cfc-4330-9af4-d2630c93db00 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:56.096834 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6696af-2cfc-4330-9af4-d2630c93db00 None admin] PATH_INFO: `/v3/roles/member` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:56.105371 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd751b2-69fd-4fd4-be1c-6bc335282592 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:56.116379 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc6696af-2cfc-4330-9af4-d2630c93db00 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:38:56.123558 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-dc6696af-2cfc-4330-9af4-d2630c93db00 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 11 00:38:56.123558 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:38:56.123558 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:38:56.123558 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:38:56.123558 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:38:56.123558 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:38:56.123558 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:38:56.123558 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:38:56.123558 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:38:56.123558 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:38:56.123558 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:38:56.123558 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:38:56.123558 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:38:56.123558 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:38:56.123558 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:38:56.123558 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:38:56.124467 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:38:56.124467 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:38:56.124467 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:38:56.124467 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:38:56.124467 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:38:56.124467 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:38:56.124467 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:38:56.124467 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:38:56.124467 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:38:56.124467 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:38:56.124467 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:38:56.124467 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:38:56.124467 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:38:56.124467 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:38:56.124467 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:38:56.124467 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:38:56.125432 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:38:56.125432 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:38:56.125432 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:38:56.125432 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:38:56.125432 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:38:56.125432 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:38:56.125432 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:38:56.125432 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:38:56.125432 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:38:56.125432 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:38:56.125432 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 11 00:38:56.125432 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:38:56.125432 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 122/247] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:38:55 2022] GET /identity/v3/roles/member => generated 84 bytes in 262 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:38:56.130530 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc6696af-2cfc-4330-9af4-d2630c93db00 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:56.149386 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:56.151734 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac5456d-cd68-4275-99fc-512dbde8efbb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:56.152787 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac5456d-cd68-4275-99fc-512dbde8efbb None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:56.153221 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac5456d-cd68-4275-99fc-512dbde8efbb None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:56.153697 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac5456d-cd68-4275-99fc-512dbde8efbb None admin] PATH_INFO: `/v3/projects/a3af44366c3a450994b868f0b5428c43` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:56.156261 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bac5456d-cd68-4275-99fc-512dbde8efbb None admin] RBAC: Authorizing `identity:get_project(project_id=a3af44366c3a450994b868f0b5428c43)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:56.158020 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bac5456d-cd68-4275-99fc-512dbde8efbb None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:56.160942 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 126/248] 10.222.0.27 () {56 vars in 1186 bytes} [Sun Sep 11 00:38:55 2022] GET /identity/v3/projects/a3af44366c3a450994b868f0b5428c43 => generated 302 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:56.167872 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac5456d-cd68-4275-99fc-512dbde8efbb None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:56.207925 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:56.285896 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc6696af-2cfc-4330-9af4-d2630c93db00 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:56.302304 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac5456d-cd68-4275-99fc-512dbde8efbb None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:56.334086 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-439719a4-f2fc-4ca6-9bf2-c04caa1071b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:56.334704 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439719a4-f2fc-4ca6-9bf2-c04caa1071b0 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:56.335035 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439719a4-f2fc-4ca6-9bf2-c04caa1071b0 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:56.335306 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439719a4-f2fc-4ca6-9bf2-c04caa1071b0 None admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:56.335972 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-439719a4-f2fc-4ca6-9bf2-c04caa1071b0 None admin] RBAC: Adding query filter params (name=member) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:38:56.336266 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-439719a4-f2fc-4ca6-9bf2-c04caa1071b0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:56.338504 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-439719a4-f2fc-4ca6-9bf2-c04caa1071b0 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:56.348922 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 123/249] 10.222.0.27 () {56 vars in 1101 bytes} [Sun Sep 11 00:38:56 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:38:56.350120 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94b81914-beff-4ecf-a92c-d957c67dbebf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:56.350582 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872be3a0-80f5-40b6-a70f-e2cf4818de65 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:56.350582 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872be3a0-80f5-40b6-a70f-e2cf4818de65 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:56.350736 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b81914-beff-4ecf-a92c-d957c67dbebf None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:56.350857 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872be3a0-80f5-40b6-a70f-e2cf4818de65 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:56.351066 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b81914-beff-4ecf-a92c-d957c67dbebf None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:56.351450 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b81914-beff-4ecf-a92c-d957c67dbebf None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:56.352187 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94b81914-beff-4ecf-a92c-d957c67dbebf None admin] RBAC: Adding query filter params (role.id=7d473d5a7be94698b3b0ed9a2f499a56, scope.project.id=a3af44366c3a450994b868f0b5428c43, user.id=0bb1412f97544e96bcaa961924138d03) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:38:56.352495 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94b81914-beff-4ecf-a92c-d957c67dbebf None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:56.353921 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 124/250] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:38:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:38:56.354533 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94b81914-beff-4ecf-a92c-d957c67dbebf None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:56.354711 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-872be3a0-80f5-40b6-a70f-e2cf4818de65 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:56.364820 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 127/251] 10.222.0.27 () {56 vars in 1374 bytes} [Sun Sep 11 00:38:56 2022] GET /identity/v3/role_assignments?user.id=0bb1412f97544e96bcaa961924138d03&scope.project.id=a3af44366c3a450994b868f0b5428c43&role.id=7d473d5a7be94698b3b0ed9a2f499a56 => generated 259 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:56.365895 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2d00f1-5680-4dca-b274-3de706ccb21e None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:56.366179 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2d00f1-5680-4dca-b274-3de706ccb21e None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:56.366503 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2d00f1-5680-4dca-b274-3de706ccb21e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:56.393435 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-1a2d00f1-5680-4dca-b274-3de706ccb21e None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:38:56.394657 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-1a2d00f1-5680-4dca-b274-3de706ccb21e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:38:56.459808 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:56.508364 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 128/252] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:38:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:38:56.520361 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-872be3a0-80f5-40b6-a70f-e2cf4818de65 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:56.561186 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45b02682-bdc2-466c-9059-e22fad7b8780 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:56.561849 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b02682-bdc2-466c-9059-e22fad7b8780 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:56.562126 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b02682-bdc2-466c-9059-e22fad7b8780 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:56.562399 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b02682-bdc2-466c-9059-e22fad7b8780 None admin] PATH_INFO: `/v3/users/e27b22a7b6a643b492a02622cf3fde8a` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:56.563612 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45b02682-bdc2-466c-9059-e22fad7b8780 None admin] RBAC: Authorizing `identity:get_user(user_id=e27b22a7b6a643b492a02622cf3fde8a)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:56.565103 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45b02682-bdc2-466c-9059-e22fad7b8780 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:56.566936 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 125/253] 10.222.0.27 () {56 vars in 1177 bytes} [Sun Sep 11 00:38:56 2022] GET /identity/v3/users/e27b22a7b6a643b492a02622cf3fde8a => generated 297 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:56.573486 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a2d00f1-5680-4dca-b274-3de706ccb21e None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:56.603438 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:56.692825 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9879a69-ae51-4638-942f-7d7045940180 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:56.693006 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9879a69-ae51-4638-942f-7d7045940180 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:56.693318 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9879a69-ae51-4638-942f-7d7045940180 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:56.694825 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 126/254] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:38:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 11 00:38:56.698610 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a2d00f1-5680-4dca-b274-3de706ccb21e None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:56.700723 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd5b2f0-fc30-46e4-bd7f-8ff531339268 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:56.701005 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd5b2f0-fc30-46e4-bd7f-8ff531339268 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:56.701273 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd5b2f0-fc30-46e4-bd7f-8ff531339268 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:56.723214 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-0cd5b2f0-fc30-46e4-bd7f-8ff531339268 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:38:56.724697 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-0cd5b2f0-fc30-46e4-bd7f-8ff531339268 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:38:56.766127 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1817fb95-f864-4005-bcd0-e8f44ad35c78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:56.766800 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1817fb95-f864-4005-bcd0-e8f44ad35c78 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:56.767082 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1817fb95-f864-4005-bcd0-e8f44ad35c78 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:56.767355 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1817fb95-f864-4005-bcd0-e8f44ad35c78 None admin] PATH_INFO: `/v3/projects/a3af44366c3a450994b868f0b5428c43` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:56.771204 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1817fb95-f864-4005-bcd0-e8f44ad35c78 None admin] RBAC: Authorizing `identity:get_project(project_id=a3af44366c3a450994b868f0b5428c43)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:56.772681 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1817fb95-f864-4005-bcd0-e8f44ad35c78 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:56.774614 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 129/255] 10.222.0.27 () {56 vars in 1186 bytes} [Sun Sep 11 00:38:56 2022] GET /identity/v3/projects/a3af44366c3a450994b868f0b5428c43 => generated 302 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:56.780210 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1817fb95-f864-4005-bcd0-e8f44ad35c78 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:56.805484 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 127/256] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:38:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 11 00:38:56.850020 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd5b2f0-fc30-46e4-bd7f-8ff531339268 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:56.961907 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:56.974876 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:56.980889 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1817fb95-f864-4005-bcd0-e8f44ad35c78 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:57.032534 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd5b2f0-fc30-46e4-bd7f-8ff531339268 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:57.047806 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8410bac7-61d8-4635-a936-3f43adfc4ff7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:57.048526 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8410bac7-61d8-4635-a936-3f43adfc4ff7 None admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:57.048809 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8410bac7-61d8-4635-a936-3f43adfc4ff7 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:57.049082 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8410bac7-61d8-4635-a936-3f43adfc4ff7 None admin] PATH_INFO: `/v3/projects/a3af44366c3a450994b868f0b5428c43/users/e27b22a7b6a643b492a02622cf3fde8a/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:57.052252 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8410bac7-61d8-4635-a936-3f43adfc4ff7 None admin] RBAC: Authorizing `identity:create_grant(project_id=a3af44366c3a450994b868f0b5428c43, user_id=e27b22a7b6a643b492a02622cf3fde8a, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:57.053704 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8410bac7-61d8-4635-a936-3f43adfc4ff7 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:57.078774 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 130/257] 10.222.0.27 () {58 vars in 1439 bytes} [Sun Sep 11 00:38:56 2022] PUT /identity/v3/projects/a3af44366c3a450994b868f0b5428c43/users/e27b22a7b6a643b492a02622cf3fde8a/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 299 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:38:57.096441 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b67935a-8ada-468b-814d-0d6d8b978d91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:57.097153 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b67935a-8ada-468b-814d-0d6d8b978d91 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:57.097482 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b67935a-8ada-468b-814d-0d6d8b978d91 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:57.097760 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b67935a-8ada-468b-814d-0d6d8b978d91 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:57.119192 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b67935a-8ada-468b-814d-0d6d8b978d91 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:38:57.124399 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-8b67935a-8ada-468b-814d-0d6d8b978d91 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:38:57.124399 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:38:57.124399 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:38:57.124399 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:38:57.124399 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:38:57.124399 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:38:57.124399 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:38:57.124399 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:38:57.124399 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:38:57.124399 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:38:57.124399 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:38:57.124399 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:38:57.124399 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:38:57.124399 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:38:57.124399 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:38:57.124399 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:38:57.125279 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:38:57.125279 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:38:57.125279 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:38:57.125279 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:38:57.125279 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:38:57.125279 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:38:57.125279 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:38:57.125279 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:38:57.125279 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:38:57.125279 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:38:57.125279 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:38:57.125279 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:38:57.125279 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:38:57.125279 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:38:57.125279 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:38:57.125279 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:38:57.126068 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:38:57.126068 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:38:57.126068 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:38:57.126068 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:38:57.126068 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:38:57.126068 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:38:57.126068 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:38:57.126068 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:38:57.126068 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:38:57.126068 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:38:57.126068 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:38:57.126068 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:38:57.126068 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 128/258] 10.222.0.27 () {56 vars in 1096 bytes} [Sun Sep 11 00:38:56 2022] GET /identity/v3/roles/admin => generated 83 bytes in 277 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:38:57.129811 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8410bac7-61d8-4635-a936-3f43adfc4ff7 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:57.133075 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b67935a-8ada-468b-814d-0d6d8b978d91 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:57.240632 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8410bac7-61d8-4635-a936-3f43adfc4ff7 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:57.279653 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b480f8f-2c83-4a24-9bcd-a05f036cc9a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:57.280735 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b480f8f-2c83-4a24-9bcd-a05f036cc9a6 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:57.281222 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b480f8f-2c83-4a24-9bcd-a05f036cc9a6 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:57.281706 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b480f8f-2c83-4a24-9bcd-a05f036cc9a6 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:57.294990 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b480f8f-2c83-4a24-9bcd-a05f036cc9a6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:38:57.300529 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-8b480f8f-2c83-4a24-9bcd-a05f036cc9a6 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 11 00:38:57.300529 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:38:57.300529 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:38:57.300529 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:38:57.300529 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:38:57.300529 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:38:57.300529 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:38:57.300529 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:38:57.300529 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:38:57.300529 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:38:57.300529 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:38:57.300529 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:38:57.300529 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:38:57.300529 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:38:57.300529 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:38:57.300529 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:38:57.301453 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:38:57.301453 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:38:57.301453 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:38:57.301453 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:38:57.301453 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:38:57.301453 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:38:57.301453 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:38:57.301453 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:38:57.301453 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:38:57.301453 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:38:57.301453 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:38:57.301453 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:38:57.301453 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:38:57.301453 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:38:57.301453 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:38:57.301453 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:38:57.302333 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:38:57.302333 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:38:57.302333 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:38:57.302333 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:38:57.302333 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:38:57.302333 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:38:57.302333 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:38:57.302333 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:38:57.302333 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:38:57.302333 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:38:57.302333 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 11 00:38:57.302333 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:38:57.304829 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 131/259] 10.222.0.27 () {56 vars in 1114 bytes} [Sun Sep 11 00:38:57 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:38:57.311399 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b480f8f-2c83-4a24-9bcd-a05f036cc9a6 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:57.331200 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:57.332864 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b67935a-8ada-468b-814d-0d6d8b978d91 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:57.396566 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:57.408496 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db3e8db0-004c-4ea7-b6ec-dd30f58e2dcd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:57.409120 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3e8db0-004c-4ea7-b6ec-dd30f58e2dcd None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:57.409407 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3e8db0-004c-4ea7-b6ec-dd30f58e2dcd None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:57.409679 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3e8db0-004c-4ea7-b6ec-dd30f58e2dcd None admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:57.410623 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db3e8db0-004c-4ea7-b6ec-dd30f58e2dcd None admin] RBAC: Adding query filter params (name=admin) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:38:57.410974 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db3e8db0-004c-4ea7-b6ec-dd30f58e2dcd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:57.412340 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db3e8db0-004c-4ea7-b6ec-dd30f58e2dcd None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:57.430389 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 129/260] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:38:57 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 298 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:38:57.430993 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b480f8f-2c83-4a24-9bcd-a05f036cc9a6 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:57.436466 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db3e8db0-004c-4ea7-b6ec-dd30f58e2dcd None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:57.471810 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1025f767-47de-4501-bf77-1ba63fa2a247 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:57.472429 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1025f767-47de-4501-bf77-1ba63fa2a247 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:57.472704 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1025f767-47de-4501-bf77-1ba63fa2a247 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:57.472969 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1025f767-47de-4501-bf77-1ba63fa2a247 None admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:57.473646 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1025f767-47de-4501-bf77-1ba63fa2a247 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:38:57.473950 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1025f767-47de-4501-bf77-1ba63fa2a247 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:57.475397 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1025f767-47de-4501-bf77-1ba63fa2a247 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:57.488961 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 132/261] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:38:57 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:38:57.496276 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1025f767-47de-4501-bf77-1ba63fa2a247 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:57.642335 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db3e8db0-004c-4ea7-b6ec-dd30f58e2dcd None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:57.655169 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:57.696788 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f0966f-7b6b-4a4c-93df-f27651f93471 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:57.697452 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f0966f-7b6b-4a4c-93df-f27651f93471 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:57.697734 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f0966f-7b6b-4a4c-93df-f27651f93471 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:57.698007 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f0966f-7b6b-4a4c-93df-f27651f93471 None admin] PATH_INFO: `/v3/users/f4b18ce48c9e4f2196e1fa77aec45f6f` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:57.699212 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40f0966f-7b6b-4a4c-93df-f27651f93471 None admin] RBAC: Authorizing `identity:get_user(user_id=f4b18ce48c9e4f2196e1fa77aec45f6f)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:57.700671 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40f0966f-7b6b-4a4c-93df-f27651f93471 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:57.702277 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 130/262] 10.222.0.27 () {56 vars in 1177 bytes} [Sun Sep 11 00:38:57 2022] GET /identity/v3/users/f4b18ce48c9e4f2196e1fa77aec45f6f => generated 283 bytes in 266 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:38:57.707774 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f0966f-7b6b-4a4c-93df-f27651f93471 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:57.717443 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1025f767-47de-4501-bf77-1ba63fa2a247 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:57.805173 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:57.835523 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b3fd232-c737-4011-8223-f6cb6f90093c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:57.836181 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3fd232-c737-4011-8223-f6cb6f90093c None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:57.836464 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3fd232-c737-4011-8223-f6cb6f90093c None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:57.836746 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3fd232-c737-4011-8223-f6cb6f90093c None admin] PATH_INFO: `/v3/users/f4b18ce48c9e4f2196e1fa77aec45f6f` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:57.843516 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b3fd232-c737-4011-8223-f6cb6f90093c None admin] RBAC: Authorizing `identity:get_user(user_id=f4b18ce48c9e4f2196e1fa77aec45f6f)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:57.844999 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b3fd232-c737-4011-8223-f6cb6f90093c None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:57.846947 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 133/263] 10.222.0.27 () {56 vars in 1177 bytes} [Sun Sep 11 00:38:57 2022] GET /identity/v3/users/f4b18ce48c9e4f2196e1fa77aec45f6f => generated 283 bytes in 353 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:57.852901 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b3fd232-c737-4011-8223-f6cb6f90093c None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:57.867328 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f0966f-7b6b-4a4c-93df-f27651f93471 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:57.904757 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d1b3224-4bd6-4b83-bc26-fa4caf2bfee2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:57.905376 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1b3224-4bd6-4b83-bc26-fa4caf2bfee2 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:57.905659 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1b3224-4bd6-4b83-bc26-fa4caf2bfee2 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:57.905934 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1b3224-4bd6-4b83-bc26-fa4caf2bfee2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:57.981319 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d1b3224-4bd6-4b83-bc26-fa4caf2bfee2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:57.982854 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d1b3224-4bd6-4b83-bc26-fa4caf2bfee2 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:58.060863 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:58.110493 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 131/264] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:38:57 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 403 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 11 00:38:58.119827 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d1b3224-4bd6-4b83-bc26-fa4caf2bfee2 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:58.171572 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b3fd232-c737-4011-8223-f6cb6f90093c None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:58.230805 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa7b73af-52c6-476d-a5d5-441faf44b715 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:58.231440 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7b73af-52c6-476d-a5d5-441faf44b715 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:58.231722 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7b73af-52c6-476d-a5d5-441faf44b715 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:58.232132 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7b73af-52c6-476d-a5d5-441faf44b715 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:58.251469 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:58.338297 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d1b3224-4bd6-4b83-bc26-fa4caf2bfee2 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:58.353331 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa7b73af-52c6-476d-a5d5-441faf44b715 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:58.355702 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa7b73af-52c6-476d-a5d5-441faf44b715 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:58.419702 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-069b4be9-3fe4-4343-ac91-68b1fc4d4c62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:58.420716 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-069b4be9-3fe4-4343-ac91-68b1fc4d4c62 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:58.421223 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-069b4be9-3fe4-4343-ac91-68b1fc4d4c62 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:58.421794 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-069b4be9-3fe4-4343-ac91-68b1fc4d4c62 None admin] PATH_INFO: `/v3/projects/a3af44366c3a450994b868f0b5428c43` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:58.425000 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-069b4be9-3fe4-4343-ac91-68b1fc4d4c62 None admin] RBAC: Authorizing `identity:get_project(project_id=a3af44366c3a450994b868f0b5428c43)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:58.426673 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-069b4be9-3fe4-4343-ac91-68b1fc4d4c62 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:58.428542 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 132/265] 10.222.0.27 () {56 vars in 1186 bytes} [Sun Sep 11 00:38:58 2022] GET /identity/v3/projects/a3af44366c3a450994b868f0b5428c43 => generated 302 bytes in 309 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:58.435317 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-069b4be9-3fe4-4343-ac91-68b1fc4d4c62 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:58.503888 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:58.516485 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 134/266] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:38:57 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 664 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 11 00:38:58.530615 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa7b73af-52c6-476d-a5d5-441faf44b715 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:58.601510 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:58.632754 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-069b4be9-3fe4-4343-ac91-68b1fc4d4c62 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:58.698310 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68db6792-c278-4cd1-8848-ba5939770f63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:58.700642 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68db6792-c278-4cd1-8848-ba5939770f63 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:58.701356 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68db6792-c278-4cd1-8848-ba5939770f63 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:58.702564 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68db6792-c278-4cd1-8848-ba5939770f63 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:58.704064 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68db6792-c278-4cd1-8848-ba5939770f63 None admin] RBAC: Adding query filter params (role.id=7d9213c1187c4b5182bf8132cf23ea19, scope.project.id=a3af44366c3a450994b868f0b5428c43, user.id=f4b18ce48c9e4f2196e1fa77aec45f6f) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:38:58.704748 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68db6792-c278-4cd1-8848-ba5939770f63 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:58.706742 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa7b73af-52c6-476d-a5d5-441faf44b715 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:58.708416 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68db6792-c278-4cd1-8848-ba5939770f63 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:58.720938 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 133/267] 10.222.0.27 () {56 vars in 1374 bytes} [Sun Sep 11 00:38:58 2022] GET /identity/v3/role_assignments?user.id=f4b18ce48c9e4f2196e1fa77aec45f6f&scope.project.id=a3af44366c3a450994b868f0b5428c43&role.id=7d9213c1187c4b5182bf8132cf23ea19 => generated 608 bytes in 286 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:38:58.756184 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52b77641-e1d1-4c88-b8d0-d6ea355cbb23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:58.757211 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b77641-e1d1-4c88-b8d0-d6ea355cbb23 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:58.757689 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b77641-e1d1-4c88-b8d0-d6ea355cbb23 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:58.758153 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b77641-e1d1-4c88-b8d0-d6ea355cbb23 None admin] PATH_INFO: `/v3/projects/a3af44366c3a450994b868f0b5428c43` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:58.762359 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52b77641-e1d1-4c88-b8d0-d6ea355cbb23 None admin] RBAC: Authorizing `identity:get_project(project_id=a3af44366c3a450994b868f0b5428c43)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:58.764236 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52b77641-e1d1-4c88-b8d0-d6ea355cbb23 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:58.766627 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 135/268] 10.222.0.27 () {56 vars in 1186 bytes} [Sun Sep 11 00:38:58 2022] GET /identity/v3/projects/a3af44366c3a450994b868f0b5428c43 => generated 302 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:38:58.772940 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68db6792-c278-4cd1-8848-ba5939770f63 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:58.902972 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:58.913854 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68db6792-c278-4cd1-8848-ba5939770f63 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:38:58.951631 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ea428b8-0cea-4304-931c-2385cc5b094c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:38:58.952645 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea428b8-0cea-4304-931c-2385cc5b094c None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:58.953130 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea428b8-0cea-4304-931c-2385cc5b094c None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:58.953597 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea428b8-0cea-4304-931c-2385cc5b094c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:58.954523 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ea428b8-0cea-4304-931c-2385cc5b094c None admin] RBAC: Adding query filter params (role.id=518eab0f758241369edcdd997b98ad12, scope.project.id=a3af44366c3a450994b868f0b5428c43, user.id=f4b18ce48c9e4f2196e1fa77aec45f6f) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:38:58.955076 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ea428b8-0cea-4304-931c-2385cc5b094c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:38:58.956711 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ea428b8-0cea-4304-931c-2385cc5b094c None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:38:58.966909 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 134/269] 10.222.0.27 () {56 vars in 1374 bytes} [Sun Sep 11 00:38:58 2022] GET /identity/v3/role_assignments?user.id=f4b18ce48c9e4f2196e1fa77aec45f6f&scope.project.id=a3af44366c3a450994b868f0b5428c43&role.id=518eab0f758241369edcdd997b98ad12 => generated 608 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:38:59.524495 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5de58c-8dec-4e04-adce-90bf6e0f4d96 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:59.525528 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5de58c-8dec-4e04-adce-90bf6e0f4d96 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:59.526025 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5de58c-8dec-4e04-adce-90bf6e0f4d96 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:59.527829 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 136/270] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:38:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:38:59.537120 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d749bb3-a549-4569-8164-5997b011ead8 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:59.537708 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d749bb3-a549-4569-8164-5997b011ead8 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:59.538183 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d749bb3-a549-4569-8164-5997b011ead8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:59.548848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03cea083-20b4-4234-9208-c023ea89f721 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:59.548848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03cea083-20b4-4234-9208-c023ea89f721 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:59.548848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03cea083-20b4-4234-9208-c023ea89f721 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:59.548848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 137/271] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:38:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:38:59.557772 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7472d8-6556-4781-b458-dc15d9296d71 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:59.558391 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7472d8-6556-4781-b458-dc15d9296d71 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:59.558897 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7472d8-6556-4781-b458-dc15d9296d71 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:59.569275 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-3d749bb3-a549-4569-8164-5997b011ead8 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:38:59.570778 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-3d749bb3-a549-4569-8164-5997b011ead8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:38:59.582086 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-ca7472d8-6556-4781-b458-dc15d9296d71 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:38:59.584159 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-ca7472d8-6556-4781-b458-dc15d9296d71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:38:59.670108 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 135/272] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:38:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 11 00:38:59.675223 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 138/273] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:38:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 11 00:38:59.708225 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0307b394-4776-447e-90a3-3f922cec42e2 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:59.708738 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0307b394-4776-447e-90a3-3f922cec42e2 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:59.708738 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0307b394-4776-447e-90a3-3f922cec42e2 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:59.710109 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 136/274] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:38:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:38:59.715695 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bd23ba-dbc1-4019-a90f-f21a7104ed33 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:38:59.716192 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bd23ba-dbc1-4019-a90f-f21a7104ed33 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:38:59.716671 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bd23ba-dbc1-4019-a90f-f21a7104ed33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:38:59.735922 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-92bd23ba-dbc1-4019-a90f-f21a7104ed33 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:38:59.737472 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-92bd23ba-dbc1-4019-a90f-f21a7104ed33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:38:59.770569 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:38:59.811616 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 139/275] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:38:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:38:59.948236 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0307b394-4776-447e-90a3-3f922cec42e2 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:38:59.955303 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92bd23ba-dbc1-4019-a90f-f21a7104ed33 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:00.044356 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:00.062354 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0307b394-4776-447e-90a3-3f922cec42e2 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:00.099488 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92bd23ba-dbc1-4019-a90f-f21a7104ed33 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:00.108766 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9766dc55-677b-4d08-b661-82436ad78439 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:00.109417 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9766dc55-677b-4d08-b661-82436ad78439 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:00.109692 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9766dc55-677b-4d08-b661-82436ad78439 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:00.109956 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9766dc55-677b-4d08-b661-82436ad78439 None admin] PATH_INFO: `/v3/domains/default` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:00.111163 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9766dc55-677b-4d08-b661-82436ad78439 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:00.112639 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9766dc55-677b-4d08-b661-82436ad78439 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:00.118955 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 137/276] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:38:59 2022] GET /identity/v3/domains/default => generated 199 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:00.118955 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9766dc55-677b-4d08-b661-82436ad78439 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:00.143868 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056c0d26-4fcd-45fe-98d5-cc4eaf95df90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:00.144831 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056c0d26-4fcd-45fe-98d5-cc4eaf95df90 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:00.145377 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056c0d26-4fcd-45fe-98d5-cc4eaf95df90 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:00.145658 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056c0d26-4fcd-45fe-98d5-cc4eaf95df90 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:00.159155 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-056c0d26-4fcd-45fe-98d5-cc4eaf95df90 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:39:00.166455 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-056c0d26-4fcd-45fe-98d5-cc4eaf95df90 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 11 00:39:00.166455 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:00.166455 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:00.166455 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:00.166455 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:00.166455 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:00.166455 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:00.166455 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:00.166455 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:00.166455 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:00.166455 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:00.166455 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:00.166455 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:39:00.166455 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:39:00.166455 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:39:00.166455 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:39:00.167621 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:39:00.167621 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:39:00.167621 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:39:00.167621 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:39:00.167621 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:00.167621 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:00.167621 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:00.167621 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:00.167621 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:00.167621 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:00.167621 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:00.167621 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:00.167621 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:00.167621 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:00.167621 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:00.167621 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:00.168923 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:00.168923 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:00.168923 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:00.168923 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:00.168923 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:39:00.168923 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:39:00.168923 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:39:00.168923 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:39:00.168923 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:39:00.168923 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:39:00.168923 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 11 00:39:00.168923 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:00.168923 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 140/277] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:38:59 2022] GET /identity/v3/roles/reader => generated 84 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:39:00.171592 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056c0d26-4fcd-45fe-98d5-cc4eaf95df90 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:00.202153 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:00.224601 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9766dc55-677b-4d08-b661-82436ad78439 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:00.258550 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a09025e-1328-4faf-a753-b7e6827048bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:00.259317 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a09025e-1328-4faf-a753-b7e6827048bf None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:00.259593 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a09025e-1328-4faf-a753-b7e6827048bf None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:00.259864 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a09025e-1328-4faf-a753-b7e6827048bf None admin] PATH_INFO: `/v3/roles/member` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:00.269614 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a09025e-1328-4faf-a753-b7e6827048bf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:39:00.273899 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056c0d26-4fcd-45fe-98d5-cc4eaf95df90 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:00.274481 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-0a09025e-1328-4faf-a753-b7e6827048bf None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 11 00:39:00.274481 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:00.274481 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:00.274481 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:00.274481 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:00.274481 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:00.274481 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:00.274481 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:00.274481 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:00.274481 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:00.274481 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:00.274481 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:00.274481 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:39:00.274481 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:39:00.274481 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:39:00.274481 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:39:00.275518 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:39:00.275518 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:39:00.275518 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:39:00.275518 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:39:00.275518 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:00.275518 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:00.275518 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:00.275518 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:00.275518 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:00.275518 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:00.275518 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:00.275518 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:00.275518 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:00.275518 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:00.275518 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:00.275518 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:00.276310 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:00.276310 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:00.276310 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:00.276310 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:00.276310 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:39:00.276310 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:39:00.276310 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:39:00.276310 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:39:00.276310 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:39:00.276310 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:39:00.276310 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 11 00:39:00.276310 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:00.276310 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 138/278] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:39:00 2022] GET /identity/v3/roles/member => generated 84 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:00.276966 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a09025e-1328-4faf-a753-b7e6827048bf None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:00.292788 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:00.311144 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab970ab-4dce-49d8-8599-e16140622eff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:00.311743 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab970ab-4dce-49d8-8599-e16140622eff None admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:00.312019 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab970ab-4dce-49d8-8599-e16140622eff None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:00.312288 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab970ab-4dce-49d8-8599-e16140622eff None admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:00.312983 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab970ab-4dce-49d8-8599-e16140622eff None admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:00.314412 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab970ab-4dce-49d8-8599-e16140622eff None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:00.336845 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 141/279] 10.222.0.27 () {60 vars in 1132 bytes} [Sun Sep 11 00:39:00 2022] POST /identity/v3/users => generated 293 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:39:00.338069 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5a0775-6b2f-4283-b732-27cfc17d67b2 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:00.338351 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5a0775-6b2f-4283-b732-27cfc17d67b2 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:00.338625 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5a0775-6b2f-4283-b732-27cfc17d67b2 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:00.340236 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 142/280] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:39:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:39:00.342932 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5a0775-6b2f-4283-b732-27cfc17d67b2 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:00.404255 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a09025e-1328-4faf-a753-b7e6827048bf None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:00.440368 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:00.448558 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1354c164-bb4a-4f7b-9ede-fe57289bc4cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:00.449542 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1354c164-bb4a-4f7b-9ede-fe57289bc4cc None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:00.449846 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1354c164-bb4a-4f7b-9ede-fe57289bc4cc None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:00.450161 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1354c164-bb4a-4f7b-9ede-fe57289bc4cc None admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:00.451066 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1354c164-bb4a-4f7b-9ede-fe57289bc4cc None admin] RBAC: Adding query filter params (name=reader) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:00.451818 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1354c164-bb4a-4f7b-9ede-fe57289bc4cc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:00.453392 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1354c164-bb4a-4f7b-9ede-fe57289bc4cc None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:00.461576 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5a0775-6b2f-4283-b732-27cfc17d67b2 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:00.463534 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 139/281] 10.222.0.27 () {56 vars in 1101 bytes} [Sun Sep 11 00:39:00 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:00.464629 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-548cf2d4-3869-4645-bfb2-431124743297 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:00.464914 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-548cf2d4-3869-4645-bfb2-431124743297 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:00.465183 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-548cf2d4-3869-4645-bfb2-431124743297 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:00.485088 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-548cf2d4-3869-4645-bfb2-431124743297 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:39:00.486302 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-548cf2d4-3869-4645-bfb2-431124743297 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:39:00.504480 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20017619-431c-4f2b-b13b-4d59cf853209 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:00.505087 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20017619-431c-4f2b-b13b-4d59cf853209 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:00.505361 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20017619-431c-4f2b-b13b-4d59cf853209 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:00.505625 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20017619-431c-4f2b-b13b-4d59cf853209 None admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:00.506294 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20017619-431c-4f2b-b13b-4d59cf853209 None admin] RBAC: Adding query filter params (name=member) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:00.506588 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20017619-431c-4f2b-b13b-4d59cf853209 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:00.507997 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20017619-431c-4f2b-b13b-4d59cf853209 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:00.519112 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 143/282] 10.222.0.27 () {56 vars in 1101 bytes} [Sun Sep 11 00:39:00 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:00.519711 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20017619-431c-4f2b-b13b-4d59cf853209 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:00.565415 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 140/283] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:39:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:39:00.566564 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-548cf2d4-3869-4645-bfb2-431124743297 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:00.577178 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:00.637691 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20017619-431c-4f2b-b13b-4d59cf853209 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:00.691862 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d22b047-3abc-44aa-82e7-b3ee3985c797 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:00.692504 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d22b047-3abc-44aa-82e7-b3ee3985c797 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:00.692778 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d22b047-3abc-44aa-82e7-b3ee3985c797 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:00.693046 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d22b047-3abc-44aa-82e7-b3ee3985c797 None admin] PATH_INFO: `/v3/users/0bb1412f97544e96bcaa961924138d03` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:00.694265 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d22b047-3abc-44aa-82e7-b3ee3985c797 None admin] RBAC: Authorizing `identity:get_user(user_id=0bb1412f97544e96bcaa961924138d03)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:00.695920 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d22b047-3abc-44aa-82e7-b3ee3985c797 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:00.697962 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 144/284] 10.222.0.27 () {56 vars in 1177 bytes} [Sun Sep 11 00:39:00 2022] GET /identity/v3/users/0bb1412f97544e96bcaa961924138d03 => generated 297 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:00.703261 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d22b047-3abc-44aa-82e7-b3ee3985c797 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:00.704037 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-548cf2d4-3869-4645-bfb2-431124743297 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:00.722387 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:00.753672 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-430f1f9b-c5df-4480-ad9f-13f84bbd3125 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:00.754674 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430f1f9b-c5df-4480-ad9f-13f84bbd3125 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:00.754986 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430f1f9b-c5df-4480-ad9f-13f84bbd3125 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:00.755258 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430f1f9b-c5df-4480-ad9f-13f84bbd3125 None admin] PATH_INFO: `/v3/users/77c792bcc81441a2bfabddf620aafd22` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:00.772862 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-430f1f9b-c5df-4480-ad9f-13f84bbd3125 None admin] RBAC: Authorizing `identity:get_user(user_id=77c792bcc81441a2bfabddf620aafd22)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:00.775295 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-430f1f9b-c5df-4480-ad9f-13f84bbd3125 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:00.777501 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 141/285] 10.222.0.27 () {56 vars in 1177 bytes} [Sun Sep 11 00:39:00 2022] GET /identity/v3/users/77c792bcc81441a2bfabddf620aafd22 => generated 293 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:00.785964 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-430f1f9b-c5df-4480-ad9f-13f84bbd3125 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:00.824272 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d22b047-3abc-44aa-82e7-b3ee3985c797 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:00.861316 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9567b22a-3eed-414b-bd1b-0879f0753e98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:00.862112 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9567b22a-3eed-414b-bd1b-0879f0753e98 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:00.862488 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9567b22a-3eed-414b-bd1b-0879f0753e98 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:00.863060 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9567b22a-3eed-414b-bd1b-0879f0753e98 None admin] PATH_INFO: `/v3/projects/a3af44366c3a450994b868f0b5428c43` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:00.864124 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9567b22a-3eed-414b-bd1b-0879f0753e98 None admin] RBAC: Authorizing `identity:get_project(project_id=a3af44366c3a450994b868f0b5428c43)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:00.867092 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9567b22a-3eed-414b-bd1b-0879f0753e98 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:00.869771 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 145/286] 10.222.0.27 () {56 vars in 1186 bytes} [Sun Sep 11 00:39:00 2022] GET /identity/v3/projects/a3af44366c3a450994b868f0b5428c43 => generated 302 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:39:00.871166 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:00.871721 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9567b22a-3eed-414b-bd1b-0879f0753e98 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:00.926171 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-430f1f9b-c5df-4480-ad9f-13f84bbd3125 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:00.973289 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf5e444-32d6-4ef7-8fa5-c58189754929 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:00.974025 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf5e444-32d6-4ef7-8fa5-c58189754929 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:00.974372 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf5e444-32d6-4ef7-8fa5-c58189754929 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:00.974715 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf5e444-32d6-4ef7-8fa5-c58189754929 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:00.975565 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaf5e444-32d6-4ef7-8fa5-c58189754929 None admin] RBAC: Adding query filter params (role.id=2e64fc597b6447df99a7992d411d1a34, user.id=77c792bcc81441a2bfabddf620aafd22, scope.system=all) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:00.976421 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaf5e444-32d6-4ef7-8fa5-c58189754929 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:00.979052 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaf5e444-32d6-4ef7-8fa5-c58189754929 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:00.983686 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:00.985837 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 142/287] 10.222.0.27 () {56 vars in 1308 bytes} [Sun Sep 11 00:39:00 2022] GET /identity/v3/role_assignments?user.id=77c792bcc81441a2bfabddf620aafd22&scope.system=all&role.id=2e64fc597b6447df99a7992d411d1a34 => generated 226 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:00.991092 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf5e444-32d6-4ef7-8fa5-c58189754929 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:01.026374 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9567b22a-3eed-414b-bd1b-0879f0753e98 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:01.113103 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5530c36-3cfc-4c90-b65a-286bf40aa46b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:01.113749 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5530c36-3cfc-4c90-b65a-286bf40aa46b None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:01.114019 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5530c36-3cfc-4c90-b65a-286bf40aa46b None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:01.114291 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5530c36-3cfc-4c90-b65a-286bf40aa46b None admin] PATH_INFO: `/v3/roles/member` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:01.140211 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5530c36-3cfc-4c90-b65a-286bf40aa46b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:39:01.146607 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-e5530c36-3cfc-4c90-b65a-286bf40aa46b None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 11 00:39:01.146607 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:01.146607 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:01.146607 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:01.146607 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:01.146607 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:01.146607 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:01.146607 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:01.146607 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:01.146607 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:01.146607 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:01.146607 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:01.146607 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:39:01.146607 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:39:01.146607 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:39:01.146607 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:39:01.147603 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:39:01.147603 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:39:01.147603 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:39:01.147603 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:39:01.147603 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:01.147603 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:01.147603 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:01.147603 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:01.147603 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:01.147603 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:01.147603 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:01.147603 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:01.147603 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:01.147603 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:01.147603 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:01.147603 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:01.148420 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:01.148420 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:01.148420 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:01.148420 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:01.148420 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:39:01.148420 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:39:01.148420 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:39:01.148420 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:39:01.148420 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:39:01.148420 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:39:01.148420 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 11 00:39:01.148420 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:01.149655 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 146/288] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:39:00 2022] GET /identity/v3/roles/member => generated 84 bytes in 277 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:01.155684 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5530c36-3cfc-4c90-b65a-286bf40aa46b None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:01.194624 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf5e444-32d6-4ef7-8fa5-c58189754929 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:01.237377 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:01.250953 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe8ff846-f4fe-4b70-a8db-6a33dbd0fc46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:01.251890 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8ff846-f4fe-4b70-a8db-6a33dbd0fc46 None admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:01.252367 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8ff846-f4fe-4b70-a8db-6a33dbd0fc46 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:01.252809 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8ff846-f4fe-4b70-a8db-6a33dbd0fc46 None admin] PATH_INFO: `/v3/projects/a3af44366c3a450994b868f0b5428c43/users/0bb1412f97544e96bcaa961924138d03/roles/7d473d5a7be94698b3b0ed9a2f499a56` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:01.255146 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe8ff846-f4fe-4b70-a8db-6a33dbd0fc46 None admin] RBAC: Authorizing `identity:create_grant(project_id=a3af44366c3a450994b868f0b5428c43, user_id=0bb1412f97544e96bcaa961924138d03, role_id=7d473d5a7be94698b3b0ed9a2f499a56)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:01.257559 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe8ff846-f4fe-4b70-a8db-6a33dbd0fc46 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:01.274889 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 143/289] 10.222.0.27 () {58 vars in 1439 bytes} [Sun Sep 11 00:39:00 2022] PUT /identity/v3/projects/a3af44366c3a450994b868f0b5428c43/users/0bb1412f97544e96bcaa961924138d03/roles/7d473d5a7be94698b3b0ed9a2f499a56 => generated 0 bytes in 288 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:39:01.297894 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5530c36-3cfc-4c90-b65a-286bf40aa46b None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:01.337979 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13df373e-ae50-4ecd-990e-bb8f259b2af2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:01.338595 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13df373e-ae50-4ecd-990e-bb8f259b2af2 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:01.338897 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13df373e-ae50-4ecd-990e-bb8f259b2af2 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:01.339171 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13df373e-ae50-4ecd-990e-bb8f259b2af2 None admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:01.339843 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13df373e-ae50-4ecd-990e-bb8f259b2af2 None admin] RBAC: Adding query filter params (name=member) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:01.340143 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13df373e-ae50-4ecd-990e-bb8f259b2af2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:01.341550 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13df373e-ae50-4ecd-990e-bb8f259b2af2 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:01.351854 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 147/290] 10.222.0.27 () {56 vars in 1101 bytes} [Sun Sep 11 00:39:01 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:01.357766 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe8ff846-f4fe-4b70-a8db-6a33dbd0fc46 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:01.520324 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe8ff846-f4fe-4b70-a8db-6a33dbd0fc46 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:01.561231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d75e177f-65e3-422f-953c-01555bdd85ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:01.562249 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75e177f-65e3-422f-953c-01555bdd85ee None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:01.562724 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75e177f-65e3-422f-953c-01555bdd85ee None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:01.563230 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75e177f-65e3-422f-953c-01555bdd85ee None admin] PATH_INFO: `/v3/users/e27b22a7b6a643b492a02622cf3fde8a` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:01.564664 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d75e177f-65e3-422f-953c-01555bdd85ee None admin] RBAC: Authorizing `identity:get_user(user_id=e27b22a7b6a643b492a02622cf3fde8a)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:01.566373 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d75e177f-65e3-422f-953c-01555bdd85ee None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:01.568732 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 144/291] 10.222.0.27 () {56 vars in 1177 bytes} [Sun Sep 11 00:39:01 2022] GET /identity/v3/users/e27b22a7b6a643b492a02622cf3fde8a => generated 297 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:01.575548 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13df373e-ae50-4ecd-990e-bb8f259b2af2 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:01.582526 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:01.729195 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13df373e-ae50-4ecd-990e-bb8f259b2af2 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:01.778821 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55e312a8-80de-4369-b8b9-9eaf2f22c778 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:01.779508 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e312a8-80de-4369-b8b9-9eaf2f22c778 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:01.779785 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e312a8-80de-4369-b8b9-9eaf2f22c778 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:01.780057 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e312a8-80de-4369-b8b9-9eaf2f22c778 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:01.879728 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55e312a8-80de-4369-b8b9-9eaf2f22c778 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:01.881185 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55e312a8-80de-4369-b8b9-9eaf2f22c778 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:01.887832 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:02.020702 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 148/292] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:39:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 445 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 11 00:39:02.026331 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d75e177f-65e3-422f-953c-01555bdd85ee None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:02.171770 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:02.258615 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d75e177f-65e3-422f-953c-01555bdd85ee None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:02.327174 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0b249a-ed83-4f97-aecd-dc277d1a3c74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:02.327725 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0b249a-ed83-4f97-aecd-dc277d1a3c74 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:02.327999 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0b249a-ed83-4f97-aecd-dc277d1a3c74 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:02.328267 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0b249a-ed83-4f97-aecd-dc277d1a3c74 None admin] PATH_INFO: `/v3/projects/a3af44366c3a450994b868f0b5428c43` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:02.329381 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db0b249a-ed83-4f97-aecd-dc277d1a3c74 None admin] RBAC: Authorizing `identity:get_project(project_id=a3af44366c3a450994b868f0b5428c43)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:02.330837 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db0b249a-ed83-4f97-aecd-dc277d1a3c74 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:02.332446 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 145/293] 10.222.0.27 () {56 vars in 1186 bytes} [Sun Sep 11 00:39:02 2022] GET /identity/v3/projects/a3af44366c3a450994b868f0b5428c43 => generated 302 bytes in 307 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:02.339898 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55e312a8-80de-4369-b8b9-9eaf2f22c778 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:02.489485 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55e312a8-80de-4369-b8b9-9eaf2f22c778 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:02.495822 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:02.540070 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5168eaf1-9637-49d7-9d68-1168cb437b6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:02.540771 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5168eaf1-9637-49d7-9d68-1168cb437b6a None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:02.541046 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5168eaf1-9637-49d7-9d68-1168cb437b6a None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:02.541312 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5168eaf1-9637-49d7-9d68-1168cb437b6a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:02.542043 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5168eaf1-9637-49d7-9d68-1168cb437b6a None admin] RBAC: Adding query filter params (role.id=2e64fc597b6447df99a7992d411d1a34, scope.project.id=a3af44366c3a450994b868f0b5428c43, user.id=e27b22a7b6a643b492a02622cf3fde8a) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:02.542347 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5168eaf1-9637-49d7-9d68-1168cb437b6a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:02.543810 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5168eaf1-9637-49d7-9d68-1168cb437b6a None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:02.549792 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 149/294] 10.222.0.27 () {56 vars in 1374 bytes} [Sun Sep 11 00:39:02 2022] GET /identity/v3/role_assignments?user.id=e27b22a7b6a643b492a02622cf3fde8a&scope.project.id=a3af44366c3a450994b868f0b5428c43&role.id=2e64fc597b6447df99a7992d411d1a34 => generated 608 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:03.085860 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bfafe08-8c9f-40be-a26e-2563a87fa3fd None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:03.086367 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bfafe08-8c9f-40be-a26e-2563a87fa3fd None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:03.086443 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bfafe08-8c9f-40be-a26e-2563a87fa3fd None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:03.088106 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 146/295] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:39:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:39:03.093748 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9798614-58ae-494f-99cf-7a008cbf2fa4 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:03.094214 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9798614-58ae-494f-99cf-7a008cbf2fa4 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:03.094292 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9798614-58ae-494f-99cf-7a008cbf2fa4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:03.115074 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-c9798614-58ae-494f-99cf-7a008cbf2fa4 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:39:03.116238 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-c9798614-58ae-494f-99cf-7a008cbf2fa4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:39:03.198875 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 150/296] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:39:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:39:03.418562 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5cd816-3d5c-4c3e-baba-5485dce02bc0 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:03.418922 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5cd816-3d5c-4c3e-baba-5485dce02bc0 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:03.419097 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5cd816-3d5c-4c3e-baba-5485dce02bc0 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:03.420408 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 147/297] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:39:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:39:03.425694 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2518c9d4-90fc-4b68-80ba-776071427d7e None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:03.425984 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2518c9d4-90fc-4b68-80ba-776071427d7e None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:03.426261 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2518c9d4-90fc-4b68-80ba-776071427d7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:03.445393 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-2518c9d4-90fc-4b68-80ba-776071427d7e None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:39:03.446775 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-2518c9d4-90fc-4b68-80ba-776071427d7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:39:03.494864 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f25621-9bfc-4441-a0bb-2bf1db9eb8f3 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:03.495150 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f25621-9bfc-4441-a0bb-2bf1db9eb8f3 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:03.495422 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f25621-9bfc-4441-a0bb-2bf1db9eb8f3 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:03.497213 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 148/298] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:39:03 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:39:03.497919 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f25621-9bfc-4441-a0bb-2bf1db9eb8f3 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:03.520884 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 151/299] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:39:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:39:03.530810 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2905880-4c26-414d-abe7-161a5af27e53 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:03.530810 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2905880-4c26-414d-abe7-161a5af27e53 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:03.530810 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2905880-4c26-414d-abe7-161a5af27e53 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:03.552187 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-f2905880-4c26-414d-abe7-161a5af27e53 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:39:03.553750 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-f2905880-4c26-414d-abe7-161a5af27e53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:39:03.585401 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:03.614949 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f25621-9bfc-4441-a0bb-2bf1db9eb8f3 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:03.638306 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 152/300] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:39:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:39:03.651177 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:03.662622 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4ce8c5-1611-44d5-8edb-a69e01e7b437 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:03.663328 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4ce8c5-1611-44d5-8edb-a69e01e7b437 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:03.663604 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4ce8c5-1611-44d5-8edb-a69e01e7b437 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:03.663923 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4ce8c5-1611-44d5-8edb-a69e01e7b437 None admin] PATH_INFO: `/v3/domains/default` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:03.665150 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd4ce8c5-1611-44d5-8edb-a69e01e7b437 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:03.666910 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd4ce8c5-1611-44d5-8edb-a69e01e7b437 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:03.668872 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 149/301] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:39:03 2022] GET /identity/v3/domains/default => generated 199 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:03.676218 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2905880-4c26-414d-abe7-161a5af27e53 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:03.780991 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2905880-4c26-414d-abe7-161a5af27e53 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:03.800981 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4ce8c5-1611-44d5-8edb-a69e01e7b437 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:03.823169 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb444f2-4231-45b6-98c0-556e64c3d774 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:03.823792 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb444f2-4231-45b6-98c0-556e64c3d774 None admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:03.824067 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb444f2-4231-45b6-98c0-556e64c3d774 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:03.824331 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb444f2-4231-45b6-98c0-556e64c3d774 None admin] PATH_INFO: `/v3/groups` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:03.824962 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeb444f2-4231-45b6-98c0-556e64c3d774 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:03.826402 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeb444f2-4231-45b6-98c0-556e64c3d774 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:03.850903 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 153/302] 10.222.0.27 () {60 vars in 1134 bytes} [Sun Sep 11 00:39:03 2022] POST /identity/v3/groups => generated 229 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:39:03.912788 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807ec4fe-6fe3-4ffe-a04a-9e4272e8acce None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:03.913141 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807ec4fe-6fe3-4ffe-a04a-9e4272e8acce None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:03.913341 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807ec4fe-6fe3-4ffe-a04a-9e4272e8acce None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:03.914724 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 154/303] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:39:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 11 00:39:03.921066 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f14f68a-db02-4bda-b3ed-cde04151a9f6 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:03.921367 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f14f68a-db02-4bda-b3ed-cde04151a9f6 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:03.921561 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f14f68a-db02-4bda-b3ed-cde04151a9f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:03.938091 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4ce8c5-1611-44d5-8edb-a69e01e7b437 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:03.940098 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-4f14f68a-db02-4bda-b3ed-cde04151a9f6 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:39:03.941323 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-4f14f68a-db02-4bda-b3ed-cde04151a9f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:39:03.957148 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:03.980615 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79c4c98f-4da0-4022-b96d-19affaabb645 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:03.981562 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c4c98f-4da0-4022-b96d-19affaabb645 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:03.981894 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c4c98f-4da0-4022-b96d-19affaabb645 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:03.982162 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c4c98f-4da0-4022-b96d-19affaabb645 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:03.994858 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79c4c98f-4da0-4022-b96d-19affaabb645 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:39:03.999805 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-79c4c98f-4da0-4022-b96d-19affaabb645 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 11 00:39:03.999805 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:03.999805 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:03.999805 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:03.999805 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:03.999805 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:03.999805 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:03.999805 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:03.999805 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:03.999805 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:03.999805 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:03.999805 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:03.999805 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:39:03.999805 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:39:03.999805 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:39:03.999805 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:39:04.000755 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:39:04.000755 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:39:04.000755 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:39:04.000755 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:39:04.000755 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:04.000755 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:04.000755 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:04.000755 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:04.000755 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:04.000755 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:04.000755 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:04.000755 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:04.000755 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:04.000755 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:04.000755 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:04.000755 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:04.001557 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:04.001557 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:04.001557 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:04.001557 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:04.001557 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:39:04.001557 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:39:04.001557 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:39:04.001557 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:39:04.001557 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:39:04.001557 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:39:04.001557 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 11 00:39:04.001557 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:04.001557 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 150/304] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:39:03 2022] GET /identity/v3/roles/reader => generated 84 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:39:04.002220 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79c4c98f-4da0-4022-b96d-19affaabb645 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:04.022831 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 155/305] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:39:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 11 00:39:04.022831 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f14f68a-db02-4bda-b3ed-cde04151a9f6 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:04.083047 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:04.139663 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79c4c98f-4da0-4022-b96d-19affaabb645 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:04.148906 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f14f68a-db02-4bda-b3ed-cde04151a9f6 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:04.182639 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48160f2a-f9a8-4dc0-9acf-0d719dbf9e7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:04.183216 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48160f2a-f9a8-4dc0-9acf-0d719dbf9e7b None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:04.183488 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48160f2a-f9a8-4dc0-9acf-0d719dbf9e7b None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:04.183752 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48160f2a-f9a8-4dc0-9acf-0d719dbf9e7b None admin] PATH_INFO: `/v3/roles/member` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:04.188308 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d04b9bd2-07ee-46c6-90e5-1cfb73da8ae8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:04.189193 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d04b9bd2-07ee-46c6-90e5-1cfb73da8ae8 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:04.189467 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d04b9bd2-07ee-46c6-90e5-1cfb73da8ae8 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:04.189738 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d04b9bd2-07ee-46c6-90e5-1cfb73da8ae8 None admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:04.190412 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d04b9bd2-07ee-46c6-90e5-1cfb73da8ae8 None admin] RBAC: Adding query filter params (name=reader) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:04.190713 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d04b9bd2-07ee-46c6-90e5-1cfb73da8ae8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:04.192742 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d04b9bd2-07ee-46c6-90e5-1cfb73da8ae8 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:04.194605 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48160f2a-f9a8-4dc0-9acf-0d719dbf9e7b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:39:04.200332 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-48160f2a-f9a8-4dc0-9acf-0d719dbf9e7b None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 11 00:39:04.200332 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:04.200332 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:04.200332 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:04.200332 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:04.200332 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:04.200332 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:04.200332 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:04.200332 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:04.200332 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:04.200332 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:04.200332 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:04.200332 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:39:04.200332 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:39:04.200332 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:39:04.200332 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:39:04.201224 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:39:04.201224 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:39:04.201224 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:39:04.201224 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:39:04.201224 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:04.201224 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:04.201224 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:04.201224 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:04.201224 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:04.201224 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:04.201224 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:04.201224 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:04.201224 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:04.201224 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:04.201224 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:04.201224 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:04.202036 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:04.202036 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:04.202036 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:04.202036 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:04.202036 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:39:04.202036 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:39:04.202036 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:39:04.202036 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:39:04.202036 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:39:04.202036 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:39:04.202036 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 11 00:39:04.202036 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:04.202036 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 151/306] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:39:04 2022] GET /identity/v3/roles/member => generated 84 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:04.203067 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 156/307] 10.222.0.27 () {56 vars in 1101 bytes} [Sun Sep 11 00:39:04 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:04.207158 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48160f2a-f9a8-4dc0-9acf-0d719dbf9e7b None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:04.211757 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d04b9bd2-07ee-46c6-90e5-1cfb73da8ae8 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:04.233950 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:04.327616 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48160f2a-f9a8-4dc0-9acf-0d719dbf9e7b None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:04.341117 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:04.345302 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d04b9bd2-07ee-46c6-90e5-1cfb73da8ae8 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:04.366369 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6aae92c-2e27-4c5c-8aa7-9449dd4677aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:04.367037 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6aae92c-2e27-4c5c-8aa7-9449dd4677aa None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:04.367318 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6aae92c-2e27-4c5c-8aa7-9449dd4677aa None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:04.367586 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6aae92c-2e27-4c5c-8aa7-9449dd4677aa None admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:04.368947 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6aae92c-2e27-4c5c-8aa7-9449dd4677aa None admin] RBAC: Adding query filter params (name=member) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:04.369272 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6aae92c-2e27-4c5c-8aa7-9449dd4677aa None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:04.370758 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6aae92c-2e27-4c5c-8aa7-9449dd4677aa None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:04.379601 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e9ff05a-e4a7-4ec6-ba21-bf2a95d463af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:04.380087 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 152/308] 10.222.0.27 () {56 vars in 1101 bytes} [Sun Sep 11 00:39:04 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:04.380963 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6aae92c-2e27-4c5c-8aa7-9449dd4677aa None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:04.383658 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9ff05a-e4a7-4ec6-ba21-bf2a95d463af None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:04.385918 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9ff05a-e4a7-4ec6-ba21-bf2a95d463af None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:04.386206 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9ff05a-e4a7-4ec6-ba21-bf2a95d463af None admin] PATH_INFO: `/v3/users/d0bd51abce93473ba9ef64989ee3c0b6` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:04.403423 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e9ff05a-e4a7-4ec6-ba21-bf2a95d463af None admin] RBAC: Authorizing `identity:get_user(user_id=d0bd51abce93473ba9ef64989ee3c0b6)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:04.405443 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e9ff05a-e4a7-4ec6-ba21-bf2a95d463af None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:04.407837 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 157/309] 10.222.0.27 () {56 vars in 1177 bytes} [Sun Sep 11 00:39:04 2022] GET /identity/v3/users/d0bd51abce93473ba9ef64989ee3c0b6 => generated 293 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:04.414978 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e9ff05a-e4a7-4ec6-ba21-bf2a95d463af None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:04.486326 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:04.489648 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6aae92c-2e27-4c5c-8aa7-9449dd4677aa None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:04.522597 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e9ff05a-e4a7-4ec6-ba21-bf2a95d463af None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:04.526580 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d5789b5-0480-4f7f-a11a-31bd164fe032 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:04.527244 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5789b5-0480-4f7f-a11a-31bd164fe032 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:04.527523 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5789b5-0480-4f7f-a11a-31bd164fe032 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:04.527792 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5789b5-0480-4f7f-a11a-31bd164fe032 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:04.539129 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d5789b5-0480-4f7f-a11a-31bd164fe032 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:39:04.544604 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-9d5789b5-0480-4f7f-a11a-31bd164fe032 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 11 00:39:04.544604 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:04.544604 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:04.544604 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:04.544604 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:04.544604 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:04.544604 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:04.544604 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:04.544604 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:04.544604 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:04.544604 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:04.544604 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:04.544604 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:39:04.544604 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:39:04.544604 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:39:04.544604 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:39:04.545461 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:39:04.545461 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:39:04.545461 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:39:04.545461 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:39:04.545461 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:04.545461 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:04.545461 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:04.545461 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:04.545461 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:04.545461 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:04.545461 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:04.545461 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:04.545461 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:04.545461 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:04.545461 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:04.545461 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:04.551173 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:04.551173 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:04.551173 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:04.551173 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:04.551173 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:39:04.551173 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:39:04.551173 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:39:04.551173 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:39:04.551173 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:39:04.551173 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:39:04.551173 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 11 00:39:04.551173 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:04.551173 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 153/310] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:39:04 2022] GET /identity/v3/roles/reader => generated 84 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:04.551173 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d5789b5-0480-4f7f-a11a-31bd164fe032 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:04.567375 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d09f516-912e-4904-883a-c376cbc32a51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:04.567999 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d09f516-912e-4904-883a-c376cbc32a51 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:04.568270 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d09f516-912e-4904-883a-c376cbc32a51 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:04.568600 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d09f516-912e-4904-883a-c376cbc32a51 None admin] PATH_INFO: `/v3/users/77c792bcc81441a2bfabddf620aafd22` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:04.569708 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d09f516-912e-4904-883a-c376cbc32a51 None admin] RBAC: Authorizing `identity:get_user(user_id=77c792bcc81441a2bfabddf620aafd22)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:04.571245 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d09f516-912e-4904-883a-c376cbc32a51 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:04.573133 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 158/311] 10.222.0.27 () {56 vars in 1177 bytes} [Sun Sep 11 00:39:04 2022] GET /identity/v3/users/77c792bcc81441a2bfabddf620aafd22 => generated 293 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:04.573881 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d09f516-912e-4904-883a-c376cbc32a51 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:04.582580 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:04.659888 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d5789b5-0480-4f7f-a11a-31bd164fe032 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:04.680887 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d09f516-912e-4904-883a-c376cbc32a51 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:04.696746 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b56e2a3b-4f73-4361-ae9f-95420419333b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:04.697368 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56e2a3b-4f73-4361-ae9f-95420419333b None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:04.697638 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56e2a3b-4f73-4361-ae9f-95420419333b None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:04.697906 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56e2a3b-4f73-4361-ae9f-95420419333b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:04.698643 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b56e2a3b-4f73-4361-ae9f-95420419333b None admin] RBAC: Adding query filter params (role.id=7d473d5a7be94698b3b0ed9a2f499a56, user.id=d0bd51abce93473ba9ef64989ee3c0b6, scope.system=all) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:04.698975 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b56e2a3b-4f73-4361-ae9f-95420419333b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:04.700426 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b56e2a3b-4f73-4361-ae9f-95420419333b None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:04.714818 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 154/312] 10.222.0.27 () {56 vars in 1308 bytes} [Sun Sep 11 00:39:04 2022] GET /identity/v3/role_assignments?user.id=d0bd51abce93473ba9ef64989ee3c0b6&scope.system=all&role.id=7d473d5a7be94698b3b0ed9a2f499a56 => generated 226 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:04.714818 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b56e2a3b-4f73-4361-ae9f-95420419333b None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:04.728456 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ed0d0df-2327-4702-95e6-5d8633740bd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:04.729165 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed0d0df-2327-4702-95e6-5d8633740bd2 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:04.729539 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed0d0df-2327-4702-95e6-5d8633740bd2 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:04.729897 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed0d0df-2327-4702-95e6-5d8633740bd2 None admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:04.730665 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ed0d0df-2327-4702-95e6-5d8633740bd2 None admin] RBAC: Adding query filter params (name=reader) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:04.731108 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ed0d0df-2327-4702-95e6-5d8633740bd2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:04.732516 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ed0d0df-2327-4702-95e6-5d8633740bd2 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:04.735424 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:04.744052 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 159/313] 10.222.0.27 () {56 vars in 1101 bytes} [Sun Sep 11 00:39:04 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:04.750164 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ed0d0df-2327-4702-95e6-5d8633740bd2 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:04.832450 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:04.837017 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b56e2a3b-4f73-4361-ae9f-95420419333b None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:04.877369 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ed0d0df-2327-4702-95e6-5d8633740bd2 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:04.878320 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17ac9d49-7ceb-4bf1-9cc4-251eb741f342 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:04.879029 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ac9d49-7ceb-4bf1-9cc4-251eb741f342 None admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:04.879306 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ac9d49-7ceb-4bf1-9cc4-251eb741f342 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:04.879577 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ac9d49-7ceb-4bf1-9cc4-251eb741f342 None admin] PATH_INFO: `/v3/system/users/77c792bcc81441a2bfabddf620aafd22/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:04.881046 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17ac9d49-7ceb-4bf1-9cc4-251eb741f342 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=77c792bcc81441a2bfabddf620aafd22, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:04.882443 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17ac9d49-7ceb-4bf1-9cc4-251eb741f342 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:04.895373 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 155/314] 10.222.0.27 () {58 vars in 1334 bytes} [Sun Sep 11 00:39:04 2022] PUT /identity/v3/system/users/77c792bcc81441a2bfabddf620aafd22/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:39:04.925874 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc335e0e-5bcf-48c8-9e46-1bc137e493a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:04.926530 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc335e0e-5bcf-48c8-9e46-1bc137e493a6 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:04.926826 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc335e0e-5bcf-48c8-9e46-1bc137e493a6 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:04.927103 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc335e0e-5bcf-48c8-9e46-1bc137e493a6 None admin] PATH_INFO: `/v3/users/0bb1412f97544e96bcaa961924138d03` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:04.928334 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc335e0e-5bcf-48c8-9e46-1bc137e493a6 None admin] RBAC: Authorizing `identity:get_user(user_id=0bb1412f97544e96bcaa961924138d03)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:04.929784 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc335e0e-5bcf-48c8-9e46-1bc137e493a6 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:04.931823 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 160/315] 10.222.0.27 () {56 vars in 1177 bytes} [Sun Sep 11 00:39:04 2022] GET /identity/v3/users/0bb1412f97544e96bcaa961924138d03 => generated 297 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:04.937899 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17ac9d49-7ceb-4bf1-9cc4-251eb741f342 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:05.054774 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17ac9d49-7ceb-4bf1-9cc4-251eb741f342 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:05.097817 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae63810-b866-49b0-9402-5938369b4598 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:05.098730 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae63810-b866-49b0-9402-5938369b4598 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:05.099247 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae63810-b866-49b0-9402-5938369b4598 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:05.099689 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae63810-b866-49b0-9402-5938369b4598 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:05.135478 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:05.178309 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ae63810-b866-49b0-9402-5938369b4598 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:05.179809 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ae63810-b866-49b0-9402-5938369b4598 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:05.294776 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 156/316] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:39:04 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 357 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 11 00:39:05.303867 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc335e0e-5bcf-48c8-9e46-1bc137e493a6 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:05.379185 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:05.418227 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc335e0e-5bcf-48c8-9e46-1bc137e493a6 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:05.457860 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b869e5-11fe-463e-8952-897c2b840673 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:05.458498 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b869e5-11fe-463e-8952-897c2b840673 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:05.458776 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b869e5-11fe-463e-8952-897c2b840673 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:05.459085 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b869e5-11fe-463e-8952-897c2b840673 None admin] PATH_INFO: `/v3/projects/a3af44366c3a450994b868f0b5428c43` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:05.460218 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9b869e5-11fe-463e-8952-897c2b840673 None admin] RBAC: Authorizing `identity:get_project(project_id=a3af44366c3a450994b868f0b5428c43)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:05.461661 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9b869e5-11fe-463e-8952-897c2b840673 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:05.464093 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 161/317] 10.222.0.27 () {56 vars in 1186 bytes} [Sun Sep 11 00:39:05 2022] GET /identity/v3/projects/a3af44366c3a450994b868f0b5428c43 => generated 302 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:05.469721 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae63810-b866-49b0-9402-5938369b4598 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:05.541256 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:05.581998 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae63810-b866-49b0-9402-5938369b4598 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:05.617863 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a43f829c-a0c6-4695-a417-6ae0903d3567 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:05.618536 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43f829c-a0c6-4695-a417-6ae0903d3567 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:05.618913 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43f829c-a0c6-4695-a417-6ae0903d3567 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:05.619175 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43f829c-a0c6-4695-a417-6ae0903d3567 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:05.619908 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a43f829c-a0c6-4695-a417-6ae0903d3567 None admin] RBAC: Adding query filter params (role.id=7d473d5a7be94698b3b0ed9a2f499a56, scope.project.id=a3af44366c3a450994b868f0b5428c43, user.id=0bb1412f97544e96bcaa961924138d03) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:05.620216 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a43f829c-a0c6-4695-a417-6ae0903d3567 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:05.621677 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a43f829c-a0c6-4695-a417-6ae0903d3567 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:05.628179 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 157/318] 10.222.0.27 () {56 vars in 1374 bytes} [Sun Sep 11 00:39:05 2022] GET /identity/v3/role_assignments?user.id=0bb1412f97544e96bcaa961924138d03&scope.project.id=a3af44366c3a450994b868f0b5428c43&role.id=7d473d5a7be94698b3b0ed9a2f499a56 => generated 608 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:06.199577 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553319ce-92c9-4daf-92e3-365b6dfae1e7 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:06.199893 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553319ce-92c9-4daf-92e3-365b6dfae1e7 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:06.200127 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553319ce-92c9-4daf-92e3-365b6dfae1e7 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:06.201603 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 162/319] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:39:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:39:06.206566 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0195554-e592-476f-aa81-dfd58f8b2260 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:06.207022 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0195554-e592-476f-aa81-dfd58f8b2260 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:06.207204 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0195554-e592-476f-aa81-dfd58f8b2260 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:06.225561 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-a0195554-e592-476f-aa81-dfd58f8b2260 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:39:06.226719 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-a0195554-e592-476f-aa81-dfd58f8b2260 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:39:06.295146 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 158/320] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:39:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:39:06.569868 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-553319ce-92c9-4daf-92e3-365b6dfae1e7 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:06.674399 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-553319ce-92c9-4daf-92e3-365b6dfae1e7 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:06.709241 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-218175e2-822f-453b-bb7c-86db23896e87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:06.709867 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-218175e2-822f-453b-bb7c-86db23896e87 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:06.710140 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-218175e2-822f-453b-bb7c-86db23896e87 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:06.710405 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-218175e2-822f-453b-bb7c-86db23896e87 None admin] PATH_INFO: `/v3/domains/default` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:06.711516 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-218175e2-822f-453b-bb7c-86db23896e87 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:06.713776 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-218175e2-822f-453b-bb7c-86db23896e87 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:06.715529 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 163/321] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:39:06 2022] GET /identity/v3/domains/default => generated 199 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:06.720903 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0195554-e592-476f-aa81-dfd58f8b2260 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:06.777432 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:06.823368 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0195554-e592-476f-aa81-dfd58f8b2260 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:06.858669 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c047b62-2dfa-41f1-a029-5792fc04ea43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:06.859303 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c047b62-2dfa-41f1-a029-5792fc04ea43 None admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:06.859579 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c047b62-2dfa-41f1-a029-5792fc04ea43 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:06.859845 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c047b62-2dfa-41f1-a029-5792fc04ea43 None admin] PATH_INFO: `/v3/groups` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:06.860484 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c047b62-2dfa-41f1-a029-5792fc04ea43 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:06.861916 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c047b62-2dfa-41f1-a029-5792fc04ea43 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:06.879640 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 159/322] 10.222.0.27 () {60 vars in 1134 bytes} [Sun Sep 11 00:39:06 2022] POST /identity/v3/groups => generated 226 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:39:07.006032 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11498234-6672-4dd1-a5c9-8057e7b398c1 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:07.006361 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11498234-6672-4dd1-a5c9-8057e7b398c1 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:07.006589 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11498234-6672-4dd1-a5c9-8057e7b398c1 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:07.008182 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 164/323] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:39:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:39:07.014391 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810d4e90-4547-4184-862a-085b8059cf37 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:07.014665 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810d4e90-4547-4184-862a-085b8059cf37 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:07.014978 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810d4e90-4547-4184-862a-085b8059cf37 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:07.035801 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-810d4e90-4547-4184-862a-085b8059cf37 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:39:07.037023 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-810d4e90-4547-4184-862a-085b8059cf37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:39:07.121419 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 160/324] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:39:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:39:07.220383 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae7dc18-5456-42d6-b666-97f99539c039 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:07.220701 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae7dc18-5456-42d6-b666-97f99539c039 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:07.220941 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae7dc18-5456-42d6-b666-97f99539c039 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:07.222460 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 165/325] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:39:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:39:07.228864 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499dbd1b-71a3-462a-a90b-a0d17a462469 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:07.229751 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499dbd1b-71a3-462a-a90b-a0d17a462469 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:07.230215 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499dbd1b-71a3-462a-a90b-a0d17a462469 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:07.251771 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-499dbd1b-71a3-462a-a90b-a0d17a462469 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:39:07.253265 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-499dbd1b-71a3-462a-a90b-a0d17a462469 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:39:07.298883 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:07.327379 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 161/326] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:39:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:39:07.391453 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae7dc18-5456-42d6-b666-97f99539c039 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:07.394622 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:07.507305 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae7dc18-5456-42d6-b666-97f99539c039 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:07.543031 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f457d6-b27b-461a-b5e1-8a3f9e1f6d5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:07.544097 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f457d6-b27b-461a-b5e1-8a3f9e1f6d5c None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:07.544572 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f457d6-b27b-461a-b5e1-8a3f9e1f6d5c None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:07.545088 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f457d6-b27b-461a-b5e1-8a3f9e1f6d5c None admin] PATH_INFO: `/v3/roles/reader` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:07.558964 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24f457d6-b27b-461a-b5e1-8a3f9e1f6d5c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:39:07.564208 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-24f457d6-b27b-461a-b5e1-8a3f9e1f6d5c None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 11 00:39:07.564208 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:07.564208 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:07.564208 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:07.564208 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:07.564208 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:07.564208 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:07.564208 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:07.564208 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:07.564208 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:07.564208 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:07.564208 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:07.564208 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:39:07.564208 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:39:07.564208 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:39:07.564208 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:39:07.565132 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:39:07.565132 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:39:07.565132 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:39:07.565132 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:39:07.565132 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:07.565132 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:07.565132 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:07.565132 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:07.565132 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:07.565132 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:07.565132 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:07.565132 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:07.565132 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:07.565132 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:07.565132 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:07.565132 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:07.565937 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:07.565937 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:07.565937 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:07.565937 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:07.565937 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:39:07.565937 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:39:07.565937 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:39:07.565937 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:39:07.565937 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:39:07.565937 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:39:07.565937 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 11 00:39:07.565937 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:07.573218 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 166/327] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:39:07 2022] GET /identity/v3/roles/reader => generated 84 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:07.575360 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-499dbd1b-71a3-462a-a90b-a0d17a462469 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:07.610173 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f457d6-b27b-461a-b5e1-8a3f9e1f6d5c None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:07.684643 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:07.724089 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-499dbd1b-71a3-462a-a90b-a0d17a462469 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:07.738317 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f457d6-b27b-461a-b5e1-8a3f9e1f6d5c None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:07.762593 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a0731b-a142-480c-b818-a3790c2f3541 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:07.763470 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a0731b-a142-480c-b818-a3790c2f3541 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:07.763872 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a0731b-a142-480c-b818-a3790c2f3541 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:07.764305 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a0731b-a142-480c-b818-a3790c2f3541 None admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:07.765110 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a0731b-a142-480c-b818-a3790c2f3541 None admin] RBAC: Adding query filter params (name=reader) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:07.765548 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a0731b-a142-480c-b818-a3790c2f3541 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:07.767165 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a0731b-a142-480c-b818-a3790c2f3541 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:07.777176 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 162/328] 10.222.0.27 () {56 vars in 1101 bytes} [Sun Sep 11 00:39:07 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:07.782529 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce73123-92f2-41d1-8fbc-aa83f782772b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:07.782852 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a0731b-a142-480c-b818-a3790c2f3541 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:07.783383 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce73123-92f2-41d1-8fbc-aa83f782772b None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:07.783687 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce73123-92f2-41d1-8fbc-aa83f782772b None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:07.784042 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce73123-92f2-41d1-8fbc-aa83f782772b None admin] PATH_INFO: `/v3/roles/member` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:07.802515 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ce73123-92f2-41d1-8fbc-aa83f782772b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:39:07.808531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-4ce73123-92f2-41d1-8fbc-aa83f782772b None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 11 00:39:07.808531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:07.808531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:07.808531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:07.808531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:07.808531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:07.808531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:07.808531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:07.808531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:07.808531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:07.808531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:07.808531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:07.808531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:39:07.808531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:39:07.808531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:39:07.808531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:39:07.809630 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:39:07.809630 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:39:07.809630 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:39:07.809630 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:39:07.809630 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:07.809630 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:07.809630 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:07.809630 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:07.809630 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:07.809630 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:07.809630 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:07.809630 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:07.809630 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:07.809630 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:07.809630 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:07.809630 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:07.810598 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:07.810598 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:07.810598 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:07.810598 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:07.810598 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:39:07.810598 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:39:07.810598 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:39:07.810598 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:39:07.810598 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:39:07.810598 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:39:07.810598 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 11 00:39:07.810598 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:07.810598 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 167/329] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:39:07 2022] GET /identity/v3/roles/member => generated 84 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:39:07.817573 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce73123-92f2-41d1-8fbc-aa83f782772b None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:07.837616 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:07.929152 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a0731b-a142-480c-b818-a3790c2f3541 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:07.960816 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce73123-92f2-41d1-8fbc-aa83f782772b None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:07.972808 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73525ef1-5f73-4cd6-8524-6eda0ed2dd6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:07.973744 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73525ef1-5f73-4cd6-8524-6eda0ed2dd6e None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:07.974212 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73525ef1-5f73-4cd6-8524-6eda0ed2dd6e None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:07.974695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73525ef1-5f73-4cd6-8524-6eda0ed2dd6e None admin] PATH_INFO: `/v3/users/d0bd51abce93473ba9ef64989ee3c0b6` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:07.976043 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73525ef1-5f73-4cd6-8524-6eda0ed2dd6e None admin] RBAC: Authorizing `identity:get_user(user_id=d0bd51abce93473ba9ef64989ee3c0b6)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:07.978412 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73525ef1-5f73-4cd6-8524-6eda0ed2dd6e None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:07.980553 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 163/330] 10.222.0.27 () {56 vars in 1177 bytes} [Sun Sep 11 00:39:07 2022] GET /identity/v3/users/d0bd51abce93473ba9ef64989ee3c0b6 => generated 293 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:07.986350 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:07.987809 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73525ef1-5f73-4cd6-8524-6eda0ed2dd6e None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:08.009113 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-866b47a6-2011-402b-879c-149caea860fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:08.011099 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866b47a6-2011-402b-879c-149caea860fb None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:08.011796 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866b47a6-2011-402b-879c-149caea860fb None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:08.013441 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866b47a6-2011-402b-879c-149caea860fb None admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:08.014442 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-866b47a6-2011-402b-879c-149caea860fb None admin] RBAC: Adding query filter params (name=member) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:08.015728 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-866b47a6-2011-402b-879c-149caea860fb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:08.018535 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-866b47a6-2011-402b-879c-149caea860fb None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:08.040233 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 168/331] 10.222.0.27 () {56 vars in 1101 bytes} [Sun Sep 11 00:39:07 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:39:08.045506 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-866b47a6-2011-402b-879c-149caea860fb None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:08.189496 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73525ef1-5f73-4cd6-8524-6eda0ed2dd6e None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:08.197706 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-866b47a6-2011-402b-879c-149caea860fb None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:08.201934 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:08.245385 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e3f0ef-a5b3-470d-9a02-12d6220b2f87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:08.245945 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e3f0ef-a5b3-470d-9a02-12d6220b2f87 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:08.246219 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e3f0ef-a5b3-470d-9a02-12d6220b2f87 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:08.246491 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e3f0ef-a5b3-470d-9a02-12d6220b2f87 None admin] PATH_INFO: `/v3/users/77c792bcc81441a2bfabddf620aafd22` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:08.248425 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57e3f0ef-a5b3-470d-9a02-12d6220b2f87 None admin] RBAC: Authorizing `identity:get_user(user_id=77c792bcc81441a2bfabddf620aafd22)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:08.249884 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57e3f0ef-a5b3-470d-9a02-12d6220b2f87 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:08.251653 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 169/332] 10.222.0.27 () {56 vars in 1177 bytes} [Sun Sep 11 00:39:08 2022] GET /identity/v3/users/77c792bcc81441a2bfabddf620aafd22 => generated 293 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:08.255247 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79eba7bf-066a-4589-9523-5b15a4d101e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:08.255904 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79eba7bf-066a-4589-9523-5b15a4d101e6 None admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:08.256179 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79eba7bf-066a-4589-9523-5b15a4d101e6 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:08.256450 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79eba7bf-066a-4589-9523-5b15a4d101e6 None admin] PATH_INFO: `/v3/system/users/d0bd51abce93473ba9ef64989ee3c0b6/roles/7d473d5a7be94698b3b0ed9a2f499a56` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:08.257891 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79eba7bf-066a-4589-9523-5b15a4d101e6 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=d0bd51abce93473ba9ef64989ee3c0b6, role_id=7d473d5a7be94698b3b0ed9a2f499a56)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:08.259069 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e3f0ef-a5b3-470d-9a02-12d6220b2f87 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:08.260750 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79eba7bf-066a-4589-9523-5b15a4d101e6 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:08.278232 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 164/333] 10.222.0.27 () {58 vars in 1334 bytes} [Sun Sep 11 00:39:07 2022] PUT /identity/v3/system/users/d0bd51abce93473ba9ef64989ee3c0b6/roles/7d473d5a7be94698b3b0ed9a2f499a56 => generated 0 bytes in 291 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:39:08.330406 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:08.385468 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e3f0ef-a5b3-470d-9a02-12d6220b2f87 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:08.424215 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f223fa1-f284-4818-8adc-e8e16002b923 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:08.424828 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f223fa1-f284-4818-8adc-e8e16002b923 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:08.425100 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f223fa1-f284-4818-8adc-e8e16002b923 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:08.425369 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f223fa1-f284-4818-8adc-e8e16002b923 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:08.426100 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f223fa1-f284-4818-8adc-e8e16002b923 None admin] RBAC: Adding query filter params (role.id=2e64fc597b6447df99a7992d411d1a34, user.id=77c792bcc81441a2bfabddf620aafd22, scope.system=all) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:08.426400 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f223fa1-f284-4818-8adc-e8e16002b923 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:08.427984 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f223fa1-f284-4818-8adc-e8e16002b923 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:08.442256 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 170/334] 10.222.0.27 () {56 vars in 1308 bytes} [Sun Sep 11 00:39:08 2022] GET /identity/v3/role_assignments?user.id=77c792bcc81441a2bfabddf620aafd22&scope.system=all&role.id=2e64fc597b6447df99a7992d411d1a34 => generated 510 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:09.825068 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a80901-16dd-4587-93f8-ea0d99cab457 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:09.825611 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a80901-16dd-4587-93f8-ea0d99cab457 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:09.825611 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a80901-16dd-4587-93f8-ea0d99cab457 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:09.826981 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 165/335] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:39:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:39:09.834857 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8e98ab-c9c1-4bd0-8bc5-4be2c7523c4c None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:09.835771 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8e98ab-c9c1-4bd0-8bc5-4be2c7523c4c None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:09.836272 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8e98ab-c9c1-4bd0-8bc5-4be2c7523c4c None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:09.838620 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 171/336] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:39:09 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:39:09.842226 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95cbb42-43cd-4ada-ab5f-0beb81082cc7 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:09.842732 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95cbb42-43cd-4ada-ab5f-0beb81082cc7 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:09.843226 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95cbb42-43cd-4ada-ab5f-0beb81082cc7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:09.859184 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca97d761-aba8-4a6b-86b2-e8a269a1ce64 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:09.859782 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca97d761-aba8-4a6b-86b2-e8a269a1ce64 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:09.860265 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca97d761-aba8-4a6b-86b2-e8a269a1ce64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:09.874638 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-a95cbb42-43cd-4ada-ab5f-0beb81082cc7 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:39:09.877000 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-a95cbb42-43cd-4ada-ab5f-0beb81082cc7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:39:09.896599 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-ca97d761-aba8-4a6b-86b2-e8a269a1ce64 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:39:09.898158 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-ca97d761-aba8-4a6b-86b2-e8a269a1ce64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:39:10.015312 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 166/337] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:39:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:39:10.030865 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 172/338] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:39:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 11 00:39:10.532449 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95cbb42-43cd-4ada-ab5f-0beb81082cc7 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:10.654912 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:10.659392 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95cbb42-43cd-4ada-ab5f-0beb81082cc7 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:10.709418 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a28c8fb-c288-4f88-ac3f-56c8de3585b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:10.710067 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a28c8fb-c288-4f88-ac3f-56c8de3585b0 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:10.710341 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a28c8fb-c288-4f88-ac3f-56c8de3585b0 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:10.710608 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a28c8fb-c288-4f88-ac3f-56c8de3585b0 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:10.727080 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a28c8fb-c288-4f88-ac3f-56c8de3585b0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:39:10.732969 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-6a28c8fb-c288-4f88-ac3f-56c8de3585b0 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 11 00:39:10.732969 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:10.732969 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:10.732969 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:10.732969 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:10.732969 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:10.732969 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:10.732969 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:10.732969 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:10.732969 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:10.732969 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:10.732969 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:10.732969 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:39:10.732969 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:39:10.732969 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:39:10.732969 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:39:10.733874 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:39:10.733874 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:39:10.733874 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:39:10.733874 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:39:10.733874 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:10.733874 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:10.733874 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:10.733874 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:10.733874 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:10.733874 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:10.733874 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:10.733874 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:10.733874 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:10.733874 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:10.733874 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:10.733874 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:10.734686 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:10.734686 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:10.734686 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:10.734686 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:10.734686 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:39:10.734686 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:39:10.734686 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:39:10.734686 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:39:10.734686 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:39:10.734686 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:39:10.734686 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 11 00:39:10.734686 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:10.734686 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 167/339] 10.222.0.27 () {56 vars in 1114 bytes} [Sun Sep 11 00:39:10 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:10.739812 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca97d761-aba8-4a6b-86b2-e8a269a1ce64 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:10.743906 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a28c8fb-c288-4f88-ac3f-56c8de3585b0 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:10.809349 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:10.921979 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca97d761-aba8-4a6b-86b2-e8a269a1ce64 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:10.948639 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a28c8fb-c288-4f88-ac3f-56c8de3585b0 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:10.993296 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d9b3e6-4621-4147-829e-0bd8248f2af0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:10.993919 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d9b3e6-4621-4147-829e-0bd8248f2af0 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:10.994185 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d9b3e6-4621-4147-829e-0bd8248f2af0 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:10.994449 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d9b3e6-4621-4147-829e-0bd8248f2af0 None admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:10.995165 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d9b3e6-4621-4147-829e-0bd8248f2af0 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:10.995464 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d9b3e6-4621-4147-829e-0bd8248f2af0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:10.996852 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d9b3e6-4621-4147-829e-0bd8248f2af0 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:11.011030 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 173/340] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:39:10 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 271 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:39:11.012549 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86402dc5-dc18-4dd3-91f1-a637377a7ff1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:11.013198 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86402dc5-dc18-4dd3-91f1-a637377a7ff1 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:11.013473 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86402dc5-dc18-4dd3-91f1-a637377a7ff1 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:11.013742 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86402dc5-dc18-4dd3-91f1-a637377a7ff1 None admin] PATH_INFO: `/v3/roles/member` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:11.015472 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d9b3e6-4621-4147-829e-0bd8248f2af0 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:11.036461 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86402dc5-dc18-4dd3-91f1-a637377a7ff1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:39:11.041766 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-86402dc5-dc18-4dd3-91f1-a637377a7ff1 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 11 00:39:11.041766 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:11.041766 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:11.041766 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:11.041766 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:11.041766 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:11.041766 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:11.041766 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:11.041766 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:11.041766 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:11.041766 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:11.041766 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:11.041766 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:39:11.041766 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:39:11.041766 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:39:11.041766 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:39:11.042649 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:39:11.042649 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:39:11.042649 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:39:11.042649 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:39:11.042649 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:11.042649 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:11.042649 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:11.042649 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:11.042649 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:11.042649 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:11.042649 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:11.042649 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:11.042649 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:11.042649 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:11.042649 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:11.042649 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:11.043492 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:11.043492 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:11.043492 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:11.043492 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:11.043492 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:39:11.043492 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:39:11.043492 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:39:11.043492 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:39:11.043492 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:39:11.043492 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:39:11.043492 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 11 00:39:11.043492 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:11.043492 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 168/341] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:39:10 2022] GET /identity/v3/roles/member => generated 84 bytes in 300 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:11.048703 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86402dc5-dc18-4dd3-91f1-a637377a7ff1 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:11.096153 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:11.167587 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d9b3e6-4621-4147-829e-0bd8248f2af0 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:11.169378 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:11.230911 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65bf8a85-d82e-4aba-b6e9-aaa8436d18f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:11.231524 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65bf8a85-d82e-4aba-b6e9-aaa8436d18f0 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:11.231798 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65bf8a85-d82e-4aba-b6e9-aaa8436d18f0 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:11.232063 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65bf8a85-d82e-4aba-b6e9-aaa8436d18f0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:11.293828 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86402dc5-dc18-4dd3-91f1-a637377a7ff1 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:11.315054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65bf8a85-d82e-4aba-b6e9-aaa8436d18f0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:11.317018 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65bf8a85-d82e-4aba-b6e9-aaa8436d18f0 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:11.379854 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-005010ba-26ba-4e18-8062-b383f9a14a34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:11.380759 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005010ba-26ba-4e18-8062-b383f9a14a34 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:11.381198 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005010ba-26ba-4e18-8062-b383f9a14a34 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:11.381612 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005010ba-26ba-4e18-8062-b383f9a14a34 None admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:11.383224 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-005010ba-26ba-4e18-8062-b383f9a14a34 None admin] RBAC: Adding query filter params (name=member) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:11.386446 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-005010ba-26ba-4e18-8062-b383f9a14a34 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:11.388029 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-005010ba-26ba-4e18-8062-b383f9a14a34 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:11.408777 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 169/342] 10.222.0.27 () {56 vars in 1101 bytes} [Sun Sep 11 00:39:11 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 356 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:11.413290 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91faca4-e874-417e-9fa5-24432b4bb799 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:11.415455 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91faca4-e874-417e-9fa5-24432b4bb799 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:11.416028 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91faca4-e874-417e-9fa5-24432b4bb799 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:11.417478 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 170/343] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:39:11 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:39:11.419679 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91faca4-e874-417e-9fa5-24432b4bb799 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:11.496927 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 174/344] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:39:11 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 482 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 11 00:39:11.498206 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32a9b00-91b0-47bf-a8b2-12601f2877b3 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:11.498649 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32a9b00-91b0-47bf-a8b2-12601f2877b3 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:11.499076 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32a9b00-91b0-47bf-a8b2-12601f2877b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:11.511107 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:11.528147 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:11.531462 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-d32a9b00-91b0-47bf-a8b2-12601f2877b3 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:39:11.532631 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-d32a9b00-91b0-47bf-a8b2-12601f2877b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:39:11.554291 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91faca4-e874-417e-9fa5-24432b4bb799 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:11.594740 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2776d8a0-58a1-4cc9-b333-9ec3298183bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:11.595708 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2776d8a0-58a1-4cc9-b333-9ec3298183bd None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:11.596157 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2776d8a0-58a1-4cc9-b333-9ec3298183bd None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:11.596570 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2776d8a0-58a1-4cc9-b333-9ec3298183bd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:11.672975 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2776d8a0-58a1-4cc9-b333-9ec3298183bd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:11.674657 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2776d8a0-58a1-4cc9-b333-9ec3298183bd None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:11.709472 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 175/345] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:39:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 212 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:39:11.710385 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d32a9b00-91b0-47bf-a8b2-12601f2877b3 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:11.760855 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:11.780540 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 171/346] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:39:11 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 361 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 11 00:39:11.786556 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb3e1d1-4a10-438f-adf1-eee1f73df6e5 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:11.787069 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb3e1d1-4a10-438f-adf1-eee1f73df6e5 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:11.787570 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb3e1d1-4a10-438f-adf1-eee1f73df6e5 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:11.789074 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 172/347] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:39:11 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:39:11.790379 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eb3e1d1-4a10-438f-adf1-eee1f73df6e5 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:11.903184 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eb3e1d1-4a10-438f-adf1-eee1f73df6e5 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:11.942381 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5633e771-e200-4699-a883-65a99a4284ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:11.943412 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5633e771-e200-4699-a883-65a99a4284ab None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:11.943866 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5633e771-e200-4699-a883-65a99a4284ab None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:11.944279 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5633e771-e200-4699-a883-65a99a4284ab None admin] PATH_INFO: `/v3/projects/1cc0eb0de66448aa8d924179a50e25e9` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:11.945692 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5633e771-e200-4699-a883-65a99a4284ab None admin] RBAC: Authorizing `identity:get_project(project_id=1cc0eb0de66448aa8d924179a50e25e9)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:11.947542 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5633e771-e200-4699-a883-65a99a4284ab None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:11.949678 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 173/348] 10.222.0.27 () {56 vars in 1186 bytes} [Sun Sep 11 00:39:11 2022] GET /identity/v3/projects/1cc0eb0de66448aa8d924179a50e25e9 => generated 298 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:39:11.954862 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d32a9b00-91b0-47bf-a8b2-12601f2877b3 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:11.956857 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf11824-3850-41e3-ae6f-d2c1d09f921f None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:11.957274 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf11824-3850-41e3-ae6f-d2c1d09f921f None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:11.957767 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf11824-3850-41e3-ae6f-d2c1d09f921f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:11.978492 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-7cf11824-3850-41e3-ae6f-d2c1d09f921f None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:39:11.980064 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-7cf11824-3850-41e3-ae6f-d2c1d09f921f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:39:12.023943 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:12.037439 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:12.039133 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eca66c37-10ea-47b3-9512-87493e4974c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:12.040050 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca66c37-10ea-47b3-9512-87493e4974c1 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:12.040545 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca66c37-10ea-47b3-9512-87493e4974c1 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:12.041018 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca66c37-10ea-47b3-9512-87493e4974c1 None admin] PATH_INFO: `/v3/projects/1cc0eb0de66448aa8d924179a50e25e9` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:12.042304 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eca66c37-10ea-47b3-9512-87493e4974c1 None admin] RBAC: Authorizing `identity:get_project(project_id=1cc0eb0de66448aa8d924179a50e25e9)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:12.045113 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eca66c37-10ea-47b3-9512-87493e4974c1 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:12.047102 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 176/349] 10.222.0.27 () {56 vars in 1186 bytes} [Sun Sep 11 00:39:11 2022] GET /identity/v3/projects/1cc0eb0de66448aa8d924179a50e25e9 => generated 298 bytes in 337 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:12.047962 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eca66c37-10ea-47b3-9512-87493e4974c1 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:12.053684 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 174/350] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:39:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:39:12.055171 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76d7382-6e14-402a-82a2-5c6496fb74c5 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:12.056002 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76d7382-6e14-402a-82a2-5c6496fb74c5 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:12.056321 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76d7382-6e14-402a-82a2-5c6496fb74c5 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:12.057785 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 175/351] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:39:12 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:39:12.058571 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e76d7382-6e14-402a-82a2-5c6496fb74c5 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:12.176378 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eca66c37-10ea-47b3-9512-87493e4974c1 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:12.189520 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e76d7382-6e14-402a-82a2-5c6496fb74c5 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:12.225712 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c781116b-ee6a-401d-966e-bc055042e6f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:12.226253 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c781116b-ee6a-401d-966e-bc055042e6f3 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:12.226525 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c781116b-ee6a-401d-966e-bc055042e6f3 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:12.226815 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c781116b-ee6a-401d-966e-bc055042e6f3 None admin] PATH_INFO: `/v3/groups/3a2cf5af1fa9407ca3402e9d17116e18` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:12.235060 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1074745a-0682-4f62-826d-35c0fe12b885 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:12.235060 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1074745a-0682-4f62-826d-35c0fe12b885 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:12.235314 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1074745a-0682-4f62-826d-35c0fe12b885 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:12.235460 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1074745a-0682-4f62-826d-35c0fe12b885 None admin] PATH_INFO: `/v3/groups/3a2cf5af1fa9407ca3402e9d17116e18` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:12.237722 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c781116b-ee6a-401d-966e-bc055042e6f3 None admin] RBAC: Authorizing `identity:get_group(group_id=3a2cf5af1fa9407ca3402e9d17116e18)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:12.239239 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c781116b-ee6a-401d-966e-bc055042e6f3 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:12.241063 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 177/352] 10.222.0.27 () {56 vars in 1180 bytes} [Sun Sep 11 00:39:12 2022] GET /identity/v3/groups/3a2cf5af1fa9407ca3402e9d17116e18 => generated 226 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:12.242680 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dade9b10-1de8-4367-8775-19e90327516d None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:12.243047 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dade9b10-1de8-4367-8775-19e90327516d None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:12.243393 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dade9b10-1de8-4367-8775-19e90327516d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:12.244138 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1074745a-0682-4f62-826d-35c0fe12b885 None admin] RBAC: Authorizing `identity:get_group(group_id=3a2cf5af1fa9407ca3402e9d17116e18)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:12.245656 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1074745a-0682-4f62-826d-35c0fe12b885 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:12.247567 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 176/353] 10.222.0.27 () {56 vars in 1180 bytes} [Sun Sep 11 00:39:12 2022] GET /identity/v3/groups/3a2cf5af1fa9407ca3402e9d17116e18 => generated 226 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:12.248840 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c61b1b-edac-4a56-971f-e601528c4241 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:12.249366 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c61b1b-edac-4a56-971f-e601528c4241 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:12.249722 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c61b1b-edac-4a56-971f-e601528c4241 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:12.251441 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 177/354] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:39:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:39:12.252329 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98c61b1b-edac-4a56-971f-e601528c4241 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:12.271619 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-dade9b10-1de8-4367-8775-19e90327516d None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:39:12.272827 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-dade9b10-1de8-4367-8775-19e90327516d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:39:12.306410 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:12.356981 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:12.368408 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 178/355] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:39:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:39:12.370053 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dade9b10-1de8-4367-8775-19e90327516d None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:12.378898 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98c61b1b-edac-4a56-971f-e601528c4241 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:12.427604 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d7ae465-15ee-4581-bc10-f2be42bfc676 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:12.429156 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7ae465-15ee-4581-bc10-f2be42bfc676 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:12.429768 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7ae465-15ee-4581-bc10-f2be42bfc676 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:12.430233 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7ae465-15ee-4581-bc10-f2be42bfc676 None admin] PATH_INFO: `/v3/roles/member` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:12.446384 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d7ae465-15ee-4581-bc10-f2be42bfc676 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:39:12.451892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-7d7ae465-15ee-4581-bc10-f2be42bfc676 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 11 00:39:12.451892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:12.451892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:12.451892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:12.451892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:12.451892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:12.451892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:12.451892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:12.451892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:12.451892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:12.451892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:12.451892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:12.451892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:39:12.451892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:39:12.451892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:39:12.451892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:39:12.452861 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:39:12.452861 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:39:12.452861 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:39:12.452861 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:39:12.452861 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:12.452861 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:12.452861 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:12.452861 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:12.452861 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:12.452861 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:12.452861 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:12.452861 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:12.452861 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:12.452861 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:12.452861 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:12.452861 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:12.453707 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:12.453707 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:12.453707 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:12.453707 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:12.453707 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:39:12.453707 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:39:12.453707 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:39:12.453707 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:39:12.453707 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:39:12.453707 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:39:12.453707 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 11 00:39:12.453707 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:12.453707 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 178/356] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:39:12 2022] GET /identity/v3/roles/member => generated 84 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:12.456053 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d7ae465-15ee-4581-bc10-f2be42bfc676 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:12.501656 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dade9b10-1de8-4367-8775-19e90327516d None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:12.556028 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bdc4292-afbf-4550-b335-ad0d086c731f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:12.556929 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdc4292-afbf-4550-b335-ad0d086c731f None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:12.557200 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdc4292-afbf-4550-b335-ad0d086c731f None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:12.557461 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdc4292-afbf-4550-b335-ad0d086c731f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:12.558221 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bdc4292-afbf-4550-b335-ad0d086c731f None admin] RBAC: Adding query filter params (group.id=3a2cf5af1fa9407ca3402e9d17116e18, role.id=2e64fc597b6447df99a7992d411d1a34, scope.project.id=1cc0eb0de66448aa8d924179a50e25e9) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:12.558527 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bdc4292-afbf-4550-b335-ad0d086c731f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:12.559998 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bdc4292-afbf-4550-b335-ad0d086c731f None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:12.570856 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 179/357] 10.222.0.27 () {56 vars in 1376 bytes} [Sun Sep 11 00:39:12 2022] GET /identity/v3/role_assignments?group.id=3a2cf5af1fa9407ca3402e9d17116e18&scope.project.id=1cc0eb0de66448aa8d924179a50e25e9&role.id=2e64fc597b6447df99a7992d411d1a34 => generated 260 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:12.570856 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4cc833-3bac-4b68-b90e-2f9ab7b22ff6 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:12.570856 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4cc833-3bac-4b68-b90e-2f9ab7b22ff6 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:12.570856 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4cc833-3bac-4b68-b90e-2f9ab7b22ff6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:12.576550 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d7ae465-15ee-4581-bc10-f2be42bfc676 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:12.581009 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:12.601882 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-ff4cc833-3bac-4b68-b90e-2f9ab7b22ff6 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:39:12.603544 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-ff4cc833-3bac-4b68-b90e-2f9ab7b22ff6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:39:12.626692 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25bb7762-ed37-44e6-ae98-dacf245a028e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:12.627349 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25bb7762-ed37-44e6-ae98-dacf245a028e None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:12.627630 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25bb7762-ed37-44e6-ae98-dacf245a028e None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:12.627901 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25bb7762-ed37-44e6-ae98-dacf245a028e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:12.629247 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25bb7762-ed37-44e6-ae98-dacf245a028e None admin] RBAC: Adding query filter params (group.id=3a2cf5af1fa9407ca3402e9d17116e18, role.id=518eab0f758241369edcdd997b98ad12, scope.project.id=1cc0eb0de66448aa8d924179a50e25e9) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:12.629559 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25bb7762-ed37-44e6-ae98-dacf245a028e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:12.631495 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25bb7762-ed37-44e6-ae98-dacf245a028e None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:12.639870 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 179/358] 10.222.0.27 () {56 vars in 1376 bytes} [Sun Sep 11 00:39:12 2022] GET /identity/v3/role_assignments?group.id=3a2cf5af1fa9407ca3402e9d17116e18&scope.project.id=1cc0eb0de66448aa8d924179a50e25e9&role.id=518eab0f758241369edcdd997b98ad12 => generated 260 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:12.640767 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25bb7762-ed37-44e6-ae98-dacf245a028e None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:12.674126 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:12.694254 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 180/359] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:39:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:39:12.695270 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff4cc833-3bac-4b68-b90e-2f9ab7b22ff6 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:12.781897 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25bb7762-ed37-44e6-ae98-dacf245a028e None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:12.823462 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff4cc833-3bac-4b68-b90e-2f9ab7b22ff6 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:12.826901 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e4b4e70-8f57-4e95-8795-23b97a21687b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:12.827562 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4b4e70-8f57-4e95-8795-23b97a21687b None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:12.827844 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4b4e70-8f57-4e95-8795-23b97a21687b None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:12.828124 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4b4e70-8f57-4e95-8795-23b97a21687b None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:12.841446 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e4b4e70-8f57-4e95-8795-23b97a21687b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:39:12.848570 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-6e4b4e70-8f57-4e95-8795-23b97a21687b None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 11 00:39:12.848570 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:12.848570 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:12.848570 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:12.848570 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:12.848570 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:12.848570 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:12.848570 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:12.848570 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:12.848570 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:12.848570 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:12.848570 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:12.848570 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:39:12.848570 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:39:12.848570 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:39:12.848570 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:39:12.849551 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:39:12.849551 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:39:12.849551 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:39:12.849551 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:39:12.849551 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:12.849551 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:12.849551 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:12.849551 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:12.849551 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:12.849551 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:12.849551 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:12.849551 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:12.849551 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:12.849551 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:12.849551 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:12.849551 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:12.850364 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:12.850364 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:12.850364 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:12.850364 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:12.850364 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:39:12.850364 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:39:12.850364 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:39:12.850364 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:39:12.850364 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:39:12.850364 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:39:12.850364 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 11 00:39:12.850364 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:12.850364 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 180/360] 10.222.0.27 () {56 vars in 1114 bytes} [Sun Sep 11 00:39:12 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:12.851996 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e4b4e70-8f57-4e95-8795-23b97a21687b None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:12.878187 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9acb549-6bec-448c-8d72-b8627e20d9d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:12.878833 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9acb549-6bec-448c-8d72-b8627e20d9d2 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:12.879112 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9acb549-6bec-448c-8d72-b8627e20d9d2 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:12.879380 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9acb549-6bec-448c-8d72-b8627e20d9d2 None admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:12.880059 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9acb549-6bec-448c-8d72-b8627e20d9d2 None admin] RBAC: Adding query filter params (name=member) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:12.880354 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9acb549-6bec-448c-8d72-b8627e20d9d2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:12.881743 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9acb549-6bec-448c-8d72-b8627e20d9d2 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:12.900066 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 181/361] 10.222.0.27 () {56 vars in 1101 bytes} [Sun Sep 11 00:39:12 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:12.901065 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9acb549-6bec-448c-8d72-b8627e20d9d2 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:12.905574 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:12.979452 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:12.994282 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e4b4e70-8f57-4e95-8795-23b97a21687b None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:13.039137 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df1e8ae-7d29-4b9a-b523-7cc6f079db5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:13.040217 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df1e8ae-7d29-4b9a-b523-7cc6f079db5d None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:13.040693 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df1e8ae-7d29-4b9a-b523-7cc6f079db5d None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:13.041080 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df1e8ae-7d29-4b9a-b523-7cc6f079db5d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:13.043776 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9acb549-6bec-448c-8d72-b8627e20d9d2 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:13.055240 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2df1e8ae-7d29-4b9a-b523-7cc6f079db5d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:39:13.062312 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-2df1e8ae-7d29-4b9a-b523-7cc6f079db5d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:39:13.062312 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:13.062312 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:13.062312 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:13.062312 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:13.062312 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:13.062312 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:13.062312 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:13.062312 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:13.062312 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:13.062312 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:13.062312 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:13.062312 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:39:13.062312 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:39:13.062312 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:39:13.062312 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:39:13.063489 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:39:13.063489 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:39:13.063489 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:39:13.063489 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:39:13.063489 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:13.063489 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:13.063489 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:13.063489 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:13.063489 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:13.063489 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:13.063489 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:13.063489 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:13.063489 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:13.063489 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:13.063489 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:13.063489 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:13.064435 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:13.064435 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:13.064435 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:13.064435 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:13.064435 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:39:13.064435 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:39:13.064435 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:39:13.064435 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:39:13.064435 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:39:13.064435 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:39:13.064435 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:39:13.064435 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:13.064435 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 181/362] 10.222.0.27 () {56 vars in 1096 bytes} [Sun Sep 11 00:39:12 2022] GET /identity/v3/roles/admin => generated 83 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:13.064435 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df1e8ae-7d29-4b9a-b523-7cc6f079db5d None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:13.087714 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce1c008-9897-46dc-9fe0-685b19b46490 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:13.088326 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce1c008-9897-46dc-9fe0-685b19b46490 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:13.088600 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce1c008-9897-46dc-9fe0-685b19b46490 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:13.088865 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce1c008-9897-46dc-9fe0-685b19b46490 None admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:13.089534 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ce1c008-9897-46dc-9fe0-685b19b46490 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:13.089831 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ce1c008-9897-46dc-9fe0-685b19b46490 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:13.091243 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ce1c008-9897-46dc-9fe0-685b19b46490 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:13.110569 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 182/363] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:39:12 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:13.111637 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce1c008-9897-46dc-9fe0-685b19b46490 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:13.226525 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:13.243602 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df1e8ae-7d29-4b9a-b523-7cc6f079db5d None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:13.262271 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce1c008-9897-46dc-9fe0-685b19b46490 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:13.275977 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:13.285044 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c61e706-d407-4be0-a176-56fc2e0f397d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:13.286124 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c61e706-d407-4be0-a176-56fc2e0f397d None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:13.286580 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c61e706-d407-4be0-a176-56fc2e0f397d None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:13.287032 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c61e706-d407-4be0-a176-56fc2e0f397d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:13.302564 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-180a6761-926f-4273-952f-e1fe5a4bae5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:13.303170 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180a6761-926f-4273-952f-e1fe5a4bae5a None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:13.303543 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180a6761-926f-4273-952f-e1fe5a4bae5a None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:13.303911 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180a6761-926f-4273-952f-e1fe5a4bae5a None admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:13.304779 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-180a6761-926f-4273-952f-e1fe5a4bae5a None admin] RBAC: Adding query filter params (name=admin) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:13.305077 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-180a6761-926f-4273-952f-e1fe5a4bae5a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:13.307112 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-180a6761-926f-4273-952f-e1fe5a4bae5a None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:13.316250 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 183/364] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:39:13 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:13.317156 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-180a6761-926f-4273-952f-e1fe5a4bae5a None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:13.379475 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c61e706-d407-4be0-a176-56fc2e0f397d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:13.380984 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c61e706-d407-4be0-a176-56fc2e0f397d None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:13.442621 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-180a6761-926f-4273-952f-e1fe5a4bae5a None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:13.487794 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ce6840d-3ed2-418e-8e97-fae6ac75a10e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:13.489175 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce6840d-3ed2-418e-8e97-fae6ac75a10e None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:13.489906 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce6840d-3ed2-418e-8e97-fae6ac75a10e None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:13.490437 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce6840d-3ed2-418e-8e97-fae6ac75a10e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:13.549282 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:13.556945 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:13.574293 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 182/365] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:39:13 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 509 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 11 00:39:13.578814 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c61e706-d407-4be0-a176-56fc2e0f397d None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:13.596696 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ce6840d-3ed2-418e-8e97-fae6ac75a10e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:13.598921 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ce6840d-3ed2-418e-8e97-fae6ac75a10e None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:13.729958 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c61e706-d407-4be0-a176-56fc2e0f397d None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:13.745262 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 184/366] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:39:13 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 429 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 11 00:39:13.746411 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ce6840d-3ed2-418e-8e97-fae6ac75a10e None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:13.795227 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77af2352-6f4f-4c08-90d4-f74439789d5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:13.796594 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77af2352-6f4f-4c08-90d4-f74439789d5d None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:13.796828 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77af2352-6f4f-4c08-90d4-f74439789d5d None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:13.797289 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77af2352-6f4f-4c08-90d4-f74439789d5d None admin] PATH_INFO: `/v3/roles/reader` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:13.817409 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77af2352-6f4f-4c08-90d4-f74439789d5d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:39:13.828240 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:13.828896 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-77af2352-6f4f-4c08-90d4-f74439789d5d None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 11 00:39:13.828896 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:13.828896 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:13.828896 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:13.828896 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:13.828896 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:13.828896 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:13.828896 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:13.828896 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:13.828896 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:13.828896 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:13.828896 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:13.828896 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:39:13.828896 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:39:13.828896 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:39:13.828896 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:39:13.830512 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:39:13.830512 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:39:13.830512 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:39:13.830512 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:39:13.830512 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:13.830512 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:13.830512 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:13.830512 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:13.830512 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:13.830512 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:13.830512 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:13.830512 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:13.830512 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:13.830512 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:13.830512 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:13.830512 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:13.837745 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:13.837745 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:13.837745 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:13.837745 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:13.837745 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:39:13.837745 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:39:13.837745 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:39:13.837745 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:39:13.837745 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:39:13.837745 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:39:13.837745 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 11 00:39:13.837745 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:13.837745 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 183/367] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:39:13 2022] GET /identity/v3/roles/reader => generated 84 bytes in 255 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:13.837745 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77af2352-6f4f-4c08-90d4-f74439789d5d None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:13.901240 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ce6840d-3ed2-418e-8e97-fae6ac75a10e None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:13.907144 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:13.966331 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77af2352-6f4f-4c08-90d4-f74439789d5d None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:13.967972 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9314d988-d1d1-4109-bb38-8d2d8b4b4fbb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:13.968700 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9314d988-d1d1-4109-bb38-8d2d8b4b4fbb None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:13.968986 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9314d988-d1d1-4109-bb38-8d2d8b4b4fbb None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:13.969258 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9314d988-d1d1-4109-bb38-8d2d8b4b4fbb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:14.003359 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20595c4a-25c3-42a1-881f-8609a3ba45e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:14.004691 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20595c4a-25c3-42a1-881f-8609a3ba45e7 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:14.005186 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20595c4a-25c3-42a1-881f-8609a3ba45e7 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:14.005689 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20595c4a-25c3-42a1-881f-8609a3ba45e7 None admin] PATH_INFO: `/v3/projects/a3af44366c3a450994b868f0b5428c43` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:14.007049 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20595c4a-25c3-42a1-881f-8609a3ba45e7 None admin] RBAC: Authorizing `identity:get_project(project_id=a3af44366c3a450994b868f0b5428c43)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:14.009246 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20595c4a-25c3-42a1-881f-8609a3ba45e7 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:14.011307 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 184/368] 10.222.0.27 () {56 vars in 1186 bytes} [Sun Sep 11 00:39:13 2022] GET /identity/v3/projects/a3af44366c3a450994b868f0b5428c43 => generated 302 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:14.012363 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20595c4a-25c3-42a1-881f-8609a3ba45e7 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:14.064572 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9314d988-d1d1-4109-bb38-8d2d8b4b4fbb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:14.066060 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9314d988-d1d1-4109-bb38-8d2d8b4b4fbb None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:14.128664 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20595c4a-25c3-42a1-881f-8609a3ba45e7 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:14.145851 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:14.155768 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:14.171006 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dfcc65f-4314-4ccf-8e04-9b65cac1cc48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:14.171685 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dfcc65f-4314-4ccf-8e04-9b65cac1cc48 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:14.171969 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dfcc65f-4314-4ccf-8e04-9b65cac1cc48 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:14.172245 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dfcc65f-4314-4ccf-8e04-9b65cac1cc48 None admin] PATH_INFO: `/v3/projects/a3af44366c3a450994b868f0b5428c43` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:14.173860 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dfcc65f-4314-4ccf-8e04-9b65cac1cc48 None admin] RBAC: Authorizing `identity:get_project(project_id=a3af44366c3a450994b868f0b5428c43)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:14.175656 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dfcc65f-4314-4ccf-8e04-9b65cac1cc48 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:14.177862 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 185/369] 10.222.0.27 () {56 vars in 1186 bytes} [Sun Sep 11 00:39:14 2022] GET /identity/v3/projects/a3af44366c3a450994b868f0b5428c43 => generated 302 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:14.178530 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dfcc65f-4314-4ccf-8e04-9b65cac1cc48 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:14.189501 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 185/370] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:39:13 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 443 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 11 00:39:14.190634 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9314d988-d1d1-4109-bb38-8d2d8b4b4fbb None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:14.308601 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9314d988-d1d1-4109-bb38-8d2d8b4b4fbb None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:14.324296 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dfcc65f-4314-4ccf-8e04-9b65cac1cc48 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:14.352572 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a2d84d-f56e-4f54-ab74-a256e094c08e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:14.353259 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a2d84d-f56e-4f54-ab74-a256e094c08e None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:14.353482 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a2d84d-f56e-4f54-ab74-a256e094c08e None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:14.353752 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a2d84d-f56e-4f54-ab74-a256e094c08e None admin] PATH_INFO: `/v3/groups/3a2cf5af1fa9407ca3402e9d17116e18` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:14.354940 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a2d84d-f56e-4f54-ab74-a256e094c08e None admin] RBAC: Authorizing `identity:get_group(group_id=3a2cf5af1fa9407ca3402e9d17116e18)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:14.356615 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a2d84d-f56e-4f54-ab74-a256e094c08e None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:14.358528 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 186/371] 10.222.0.27 () {56 vars in 1180 bytes} [Sun Sep 11 00:39:14 2022] GET /identity/v3/groups/3a2cf5af1fa9407ca3402e9d17116e18 => generated 226 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:14.359473 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a2d84d-f56e-4f54-ab74-a256e094c08e None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:14.366167 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43dfdad6-b5db-490b-a746-547454d3cbc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:14.367119 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43dfdad6-b5db-490b-a746-547454d3cbc2 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:14.368126 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43dfdad6-b5db-490b-a746-547454d3cbc2 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:14.368611 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43dfdad6-b5db-490b-a746-547454d3cbc2 None admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:14.370017 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43dfdad6-b5db-490b-a746-547454d3cbc2 None admin] RBAC: Adding query filter params (name=reader) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:14.370561 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43dfdad6-b5db-490b-a746-547454d3cbc2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:14.373054 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43dfdad6-b5db-490b-a746-547454d3cbc2 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:14.384090 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 186/372] 10.222.0.27 () {56 vars in 1101 bytes} [Sun Sep 11 00:39:14 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:14.385493 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43dfdad6-b5db-490b-a746-547454d3cbc2 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:14.411472 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:14.444096 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:14.475666 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a2d84d-f56e-4f54-ab74-a256e094c08e None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:14.515946 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43dfdad6-b5db-490b-a746-547454d3cbc2 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:14.534023 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91509e7e-f0b5-4f31-a5a5-e8de14e0c1a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:14.534656 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91509e7e-f0b5-4f31-a5a5-e8de14e0c1a1 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:14.534957 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91509e7e-f0b5-4f31-a5a5-e8de14e0c1a1 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:14.535470 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91509e7e-f0b5-4f31-a5a5-e8de14e0c1a1 None admin] PATH_INFO: `/v3/groups/3a2cf5af1fa9407ca3402e9d17116e18` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:14.537936 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91509e7e-f0b5-4f31-a5a5-e8de14e0c1a1 None admin] RBAC: Authorizing `identity:get_group(group_id=3a2cf5af1fa9407ca3402e9d17116e18)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:14.538823 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91509e7e-f0b5-4f31-a5a5-e8de14e0c1a1 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:14.541421 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 187/373] 10.222.0.27 () {56 vars in 1180 bytes} [Sun Sep 11 00:39:14 2022] GET /identity/v3/groups/3a2cf5af1fa9407ca3402e9d17116e18 => generated 226 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:14.542268 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91509e7e-f0b5-4f31-a5a5-e8de14e0c1a1 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:14.563705 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c2e6fc1-4bf2-4dce-944a-987f36869f0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:14.564650 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2e6fc1-4bf2-4dce-944a-987f36869f0a None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:14.565199 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2e6fc1-4bf2-4dce-944a-987f36869f0a None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:14.565665 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2e6fc1-4bf2-4dce-944a-987f36869f0a None admin] PATH_INFO: `/v3/projects/12356f1769944cc8be5c533d062adf65` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:14.567071 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c2e6fc1-4bf2-4dce-944a-987f36869f0a None admin] RBAC: Authorizing `identity:get_project(project_id=12356f1769944cc8be5c533d062adf65)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:14.568708 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c2e6fc1-4bf2-4dce-944a-987f36869f0a None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:14.570561 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 187/374] 10.222.0.27 () {56 vars in 1186 bytes} [Sun Sep 11 00:39:14 2022] GET /identity/v3/projects/12356f1769944cc8be5c533d062adf65 => generated 344 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:14.572458 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c2e6fc1-4bf2-4dce-944a-987f36869f0a None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:14.655638 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91509e7e-f0b5-4f31-a5a5-e8de14e0c1a1 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:14.670155 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:14.681742 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c2e6fc1-4bf2-4dce-944a-987f36869f0a None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:14.696903 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c606ba8d-c46a-4b94-8939-b4a1e93c21f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:14.697728 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c606ba8d-c46a-4b94-8939-b4a1e93c21f4 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:14.697958 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:14.698322 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c606ba8d-c46a-4b94-8939-b4a1e93c21f4 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:14.698843 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c606ba8d-c46a-4b94-8939-b4a1e93c21f4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:14.699816 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c606ba8d-c46a-4b94-8939-b4a1e93c21f4 None admin] RBAC: Adding query filter params (group.id=3a2cf5af1fa9407ca3402e9d17116e18, role.id=2e64fc597b6447df99a7992d411d1a34, scope.project.id=a3af44366c3a450994b868f0b5428c43) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:14.700378 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c606ba8d-c46a-4b94-8939-b4a1e93c21f4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:14.702018 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c606ba8d-c46a-4b94-8939-b4a1e93c21f4 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:14.708358 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 188/375] 10.222.0.27 () {56 vars in 1376 bytes} [Sun Sep 11 00:39:14 2022] GET /identity/v3/role_assignments?group.id=3a2cf5af1fa9407ca3402e9d17116e18&scope.project.id=a3af44366c3a450994b868f0b5428c43&role.id=2e64fc597b6447df99a7992d411d1a34 => generated 260 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:14.709476 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c606ba8d-c46a-4b94-8939-b4a1e93c21f4 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:14.720912 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c9d15a-1f49-438b-b22c-8f49798635bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:14.721558 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c9d15a-1f49-438b-b22c-8f49798635bb None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:14.721837 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c9d15a-1f49-438b-b22c-8f49798635bb None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:14.722101 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c9d15a-1f49-438b-b22c-8f49798635bb None admin] PATH_INFO: `/v3/users/d0bd51abce93473ba9ef64989ee3c0b6` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:14.723316 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7c9d15a-1f49-438b-b22c-8f49798635bb None admin] RBAC: Authorizing `identity:get_user(user_id=d0bd51abce93473ba9ef64989ee3c0b6)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:14.724835 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7c9d15a-1f49-438b-b22c-8f49798635bb None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:14.726773 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 188/376] 10.222.0.27 () {56 vars in 1177 bytes} [Sun Sep 11 00:39:14 2022] GET /identity/v3/users/d0bd51abce93473ba9ef64989ee3c0b6 => generated 293 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:14.728165 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c9d15a-1f49-438b-b22c-8f49798635bb None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:14.826229 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c606ba8d-c46a-4b94-8939-b4a1e93c21f4 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:14.843162 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c9d15a-1f49-438b-b22c-8f49798635bb None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:14.867593 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-181cad01-7b40-4367-a012-75f551c63ccd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:14.869081 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181cad01-7b40-4367-a012-75f551c63ccd None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:14.869572 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181cad01-7b40-4367-a012-75f551c63ccd None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:14.870036 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181cad01-7b40-4367-a012-75f551c63ccd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:14.871004 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-181cad01-7b40-4367-a012-75f551c63ccd None admin] RBAC: Adding query filter params (group.id=3a2cf5af1fa9407ca3402e9d17116e18, role.id=518eab0f758241369edcdd997b98ad12, scope.project.id=a3af44366c3a450994b868f0b5428c43) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:14.871534 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-181cad01-7b40-4367-a012-75f551c63ccd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:14.873233 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-181cad01-7b40-4367-a012-75f551c63ccd None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:14.880967 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 189/377] 10.222.0.27 () {56 vars in 1376 bytes} [Sun Sep 11 00:39:14 2022] GET /identity/v3/role_assignments?group.id=3a2cf5af1fa9407ca3402e9d17116e18&scope.project.id=a3af44366c3a450994b868f0b5428c43&role.id=518eab0f758241369edcdd997b98ad12 => generated 260 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:14.881640 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f22b472-76c8-4bc8-a001-50466129b246 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:14.882246 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f22b472-76c8-4bc8-a001-50466129b246 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:14.882527 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f22b472-76c8-4bc8-a001-50466129b246 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:14.882848 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f22b472-76c8-4bc8-a001-50466129b246 None admin] PATH_INFO: `/v3/groups/dbc01bc745ca46db93dee6ac3ec2c72e` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:14.883082 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-181cad01-7b40-4367-a012-75f551c63ccd None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:14.889204 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f22b472-76c8-4bc8-a001-50466129b246 None admin] RBAC: Authorizing `identity:get_group(group_id=dbc01bc745ca46db93dee6ac3ec2c72e)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:14.890761 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f22b472-76c8-4bc8-a001-50466129b246 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:14.892856 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 189/378] 10.222.0.27 () {56 vars in 1180 bytes} [Sun Sep 11 00:39:14 2022] GET /identity/v3/groups/dbc01bc745ca46db93dee6ac3ec2c72e => generated 229 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:14.899507 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f22b472-76c8-4bc8-a001-50466129b246 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:14.932041 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:14.954681 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:15.003799 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-181cad01-7b40-4367-a012-75f551c63ccd None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:15.022811 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f22b472-76c8-4bc8-a001-50466129b246 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:15.058752 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d14695-ac21-4c31-a4bb-85a5cf620f8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:15.059649 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d14695-ac21-4c31-a4bb-85a5cf620f8e None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:15.060062 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d14695-ac21-4c31-a4bb-85a5cf620f8e None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:15.060508 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d14695-ac21-4c31-a4bb-85a5cf620f8e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:15.061449 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d14695-ac21-4c31-a4bb-85a5cf620f8e None admin] RBAC: Adding query filter params (role.id=7d473d5a7be94698b3b0ed9a2f499a56, user.id=d0bd51abce93473ba9ef64989ee3c0b6, scope.system=all) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:15.061901 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d14695-ac21-4c31-a4bb-85a5cf620f8e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:15.063529 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d14695-ac21-4c31-a4bb-85a5cf620f8e None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:15.064011 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7820a8b8-9755-4114-8133-fb99a56eb023 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:15.064715 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7820a8b8-9755-4114-8133-fb99a56eb023 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:15.065086 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7820a8b8-9755-4114-8133-fb99a56eb023 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:15.065460 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7820a8b8-9755-4114-8133-fb99a56eb023 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:15.066388 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7820a8b8-9755-4114-8133-fb99a56eb023 None admin] RBAC: Adding query filter params (group.id=dbc01bc745ca46db93dee6ac3ec2c72e, role.id=7d9213c1187c4b5182bf8132cf23ea19, scope.project.id=12356f1769944cc8be5c533d062adf65) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:15.066814 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7820a8b8-9755-4114-8133-fb99a56eb023 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:15.068494 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7820a8b8-9755-4114-8133-fb99a56eb023 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:15.078816 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 190/379] 10.222.0.27 () {56 vars in 1376 bytes} [Sun Sep 11 00:39:14 2022] GET /identity/v3/role_assignments?group.id=dbc01bc745ca46db93dee6ac3ec2c72e&scope.project.id=12356f1769944cc8be5c533d062adf65&role.id=7d9213c1187c4b5182bf8132cf23ea19 => generated 260 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:15.084009 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 190/380] 10.222.0.27 () {56 vars in 1308 bytes} [Sun Sep 11 00:39:14 2022] GET /identity/v3/role_assignments?user.id=d0bd51abce93473ba9ef64989ee3c0b6&scope.system=all&role.id=7d473d5a7be94698b3b0ed9a2f499a56 => generated 510 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:15.565679 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559001d5-b2e5-4909-840f-53a0f753edb2 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:15.566088 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559001d5-b2e5-4909-840f-53a0f753edb2 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:15.566264 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559001d5-b2e5-4909-840f-53a0f753edb2 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:15.567893 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 191/381] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:39:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:39:15.574127 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f966ac95-bdcd-4aea-8332-f4cc6bfad303 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:15.574741 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f966ac95-bdcd-4aea-8332-f4cc6bfad303 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:15.575310 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f966ac95-bdcd-4aea-8332-f4cc6bfad303 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:15.597169 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-f966ac95-bdcd-4aea-8332-f4cc6bfad303 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:39:15.598734 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-f966ac95-bdcd-4aea-8332-f4cc6bfad303 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:39:15.693610 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 191/382] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:39:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 11 00:39:15.739024 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6dca822-489c-4bf3-b3a4-e2ade4b9cab3 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:15.739371 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6dca822-489c-4bf3-b3a4-e2ade4b9cab3 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:15.739665 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6dca822-489c-4bf3-b3a4-e2ade4b9cab3 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:15.741298 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 192/383] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:39:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:39:15.748193 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-239bfa0c-0d8e-4afa-99de-72311d039ee0 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:15.748668 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-239bfa0c-0d8e-4afa-99de-72311d039ee0 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:15.749082 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-239bfa0c-0d8e-4afa-99de-72311d039ee0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:15.771305 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-239bfa0c-0d8e-4afa-99de-72311d039ee0 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:39:15.772715 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-239bfa0c-0d8e-4afa-99de-72311d039ee0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:39:15.796965 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:15.858321 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 192/384] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:39:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:39:15.997924 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6dca822-489c-4bf3-b3a4-e2ade4b9cab3 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:16.118542 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6dca822-489c-4bf3-b3a4-e2ade4b9cab3 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:16.130410 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:16.159632 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e58197-90ce-41a3-afc2-2e705ba24d4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:16.160572 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e58197-90ce-41a3-afc2-2e705ba24d4f None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:16.160859 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e58197-90ce-41a3-afc2-2e705ba24d4f None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:16.161127 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e58197-90ce-41a3-afc2-2e705ba24d4f None admin] PATH_INFO: `/v3/roles/member` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:16.174747 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78e58197-90ce-41a3-afc2-2e705ba24d4f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:39:16.179612 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-78e58197-90ce-41a3-afc2-2e705ba24d4f None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 11 00:39:16.179612 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:16.179612 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:16.179612 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:16.179612 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:16.179612 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:16.179612 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:16.179612 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:16.179612 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:16.179612 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:16.179612 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:16.179612 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:16.179612 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:39:16.179612 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:39:16.179612 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:39:16.179612 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:39:16.180573 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:39:16.180573 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:39:16.180573 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:39:16.180573 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:39:16.180573 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:16.180573 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:16.180573 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:16.180573 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:16.180573 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:16.180573 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:16.180573 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:16.180573 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:16.180573 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:16.180573 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:16.180573 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:16.180573 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:16.181571 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:16.181571 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:16.181571 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:16.181571 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:16.181571 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:39:16.181571 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:39:16.181571 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:39:16.181571 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:39:16.181571 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:39:16.181571 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:39:16.181571 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 11 00:39:16.181571 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:16.181571 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 193/385] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:39:15 2022] GET /identity/v3/roles/member => generated 84 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:16.187211 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-239bfa0c-0d8e-4afa-99de-72311d039ee0 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:16.239136 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e58197-90ce-41a3-afc2-2e705ba24d4f None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:16.357359 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-239bfa0c-0d8e-4afa-99de-72311d039ee0 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:16.397661 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e58197-90ce-41a3-afc2-2e705ba24d4f None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:16.440720 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a50ebd0-30f9-43e9-8a5d-c0fe562d39e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:16.441358 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a50ebd0-30f9-43e9-8a5d-c0fe562d39e3 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:16.441637 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a50ebd0-30f9-43e9-8a5d-c0fe562d39e3 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:16.441966 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a50ebd0-30f9-43e9-8a5d-c0fe562d39e3 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:16.451512 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e663d18-017d-4e93-8fe5-746b6b01d647 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:16.452474 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e663d18-017d-4e93-8fe5-746b6b01d647 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:16.452965 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e663d18-017d-4e93-8fe5-746b6b01d647 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:16.453157 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a50ebd0-30f9-43e9-8a5d-c0fe562d39e3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:39:16.453616 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e663d18-017d-4e93-8fe5-746b6b01d647 None admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:16.454485 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e663d18-017d-4e93-8fe5-746b6b01d647 None admin] RBAC: Adding query filter params (name=member) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:16.455153 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e663d18-017d-4e93-8fe5-746b6b01d647 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:16.456745 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e663d18-017d-4e93-8fe5-746b6b01d647 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:16.459105 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-8a50ebd0-30f9-43e9-8a5d-c0fe562d39e3 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 11 00:39:16.459105 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:16.459105 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:16.459105 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:16.459105 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:16.459105 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:16.459105 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:16.459105 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:16.459105 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:16.459105 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:16.459105 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:16.459105 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:16.459105 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:39:16.459105 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:39:16.459105 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:39:16.459105 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:39:16.460008 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:39:16.460008 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:39:16.460008 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:39:16.460008 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:39:16.460008 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:16.460008 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:16.460008 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:16.460008 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:16.460008 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:16.460008 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:16.460008 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:16.460008 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:16.460008 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:16.460008 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:16.460008 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:16.460008 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:16.460791 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:16.460791 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:16.460791 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:16.460791 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:16.460791 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:39:16.460791 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:39:16.460791 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:39:16.460791 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:39:16.460791 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:39:16.460791 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:39:16.460791 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 11 00:39:16.460791 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:16.466077 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 194/386] 10.222.0.27 () {56 vars in 1114 bytes} [Sun Sep 11 00:39:16 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:39:16.467512 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a50ebd0-30f9-43e9-8a5d-c0fe562d39e3 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:16.474822 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 193/387] 10.222.0.27 () {56 vars in 1101 bytes} [Sun Sep 11 00:39:16 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 288 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:16.500510 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e663d18-017d-4e93-8fe5-746b6b01d647 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:16.507216 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:16.565798 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:16.584121 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a50ebd0-30f9-43e9-8a5d-c0fe562d39e3 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:16.620671 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e586f0-6e56-4d4a-8d13-6d31bcb53c12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:16.621291 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e586f0-6e56-4d4a-8d13-6d31bcb53c12 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:16.621570 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e586f0-6e56-4d4a-8d13-6d31bcb53c12 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:16.621898 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e586f0-6e56-4d4a-8d13-6d31bcb53c12 None admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:16.622565 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e586f0-6e56-4d4a-8d13-6d31bcb53c12 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:16.624022 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e586f0-6e56-4d4a-8d13-6d31bcb53c12 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:16.625414 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e586f0-6e56-4d4a-8d13-6d31bcb53c12 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:16.638869 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 195/388] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:39:16 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:39:16.641040 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e586f0-6e56-4d4a-8d13-6d31bcb53c12 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:16.713694 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e663d18-017d-4e93-8fe5-746b6b01d647 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:16.751168 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e586f0-6e56-4d4a-8d13-6d31bcb53c12 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:16.774585 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c9546b-e1f1-406c-9c9d-7d82712152b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:16.775340 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c9546b-e1f1-406c-9c9d-7d82712152b8 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:16.775622 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c9546b-e1f1-406c-9c9d-7d82712152b8 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:16.775899 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c9546b-e1f1-406c-9c9d-7d82712152b8 None admin] PATH_INFO: `/v3/groups/3a2cf5af1fa9407ca3402e9d17116e18` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:16.777034 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00c9546b-e1f1-406c-9c9d-7d82712152b8 None admin] RBAC: Authorizing `identity:get_group(group_id=3a2cf5af1fa9407ca3402e9d17116e18)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:16.777808 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:16.783315 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00c9546b-e1f1-406c-9c9d-7d82712152b8 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:16.786166 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 194/389] 10.222.0.27 () {56 vars in 1180 bytes} [Sun Sep 11 00:39:16 2022] GET /identity/v3/groups/3a2cf5af1fa9407ca3402e9d17116e18 => generated 226 bytes in 285 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:16.791986 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c9546b-e1f1-406c-9c9d-7d82712152b8 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:16.798183 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a95b683-ae43-4576-9203-aed272954ce8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:16.798860 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a95b683-ae43-4576-9203-aed272954ce8 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:16.799142 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a95b683-ae43-4576-9203-aed272954ce8 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:16.799411 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a95b683-ae43-4576-9203-aed272954ce8 None admin] PATH_INFO: `/v3/groups/3a2cf5af1fa9407ca3402e9d17116e18` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:16.807407 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a95b683-ae43-4576-9203-aed272954ce8 None admin] RBAC: Authorizing `identity:get_group(group_id=3a2cf5af1fa9407ca3402e9d17116e18)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:16.808875 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a95b683-ae43-4576-9203-aed272954ce8 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:16.810490 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 196/390] 10.222.0.27 () {56 vars in 1180 bytes} [Sun Sep 11 00:39:16 2022] GET /identity/v3/groups/3a2cf5af1fa9407ca3402e9d17116e18 => generated 226 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:16.816191 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a95b683-ae43-4576-9203-aed272954ce8 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:16.963981 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:16.983842 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a95b683-ae43-4576-9203-aed272954ce8 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:16.994626 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c9546b-e1f1-406c-9c9d-7d82712152b8 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:17.045787 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92180445-2c59-488c-800b-9acaa6825eb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:17.046430 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92180445-2c59-488c-800b-9acaa6825eb6 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:17.046709 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92180445-2c59-488c-800b-9acaa6825eb6 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:17.047006 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92180445-2c59-488c-800b-9acaa6825eb6 None admin] PATH_INFO: `/v3/projects/1cc0eb0de66448aa8d924179a50e25e9` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:17.049422 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5374c990-5516-47a5-8786-e0e89be0d749 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:17.050110 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5374c990-5516-47a5-8786-e0e89be0d749 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:17.050386 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5374c990-5516-47a5-8786-e0e89be0d749 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:17.050697 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5374c990-5516-47a5-8786-e0e89be0d749 None admin] PATH_INFO: `/v3/projects/1cc0eb0de66448aa8d924179a50e25e9` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:17.051459 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92180445-2c59-488c-800b-9acaa6825eb6 None admin] RBAC: Authorizing `identity:get_project(project_id=1cc0eb0de66448aa8d924179a50e25e9)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:17.051923 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5374c990-5516-47a5-8786-e0e89be0d749 None admin] RBAC: Authorizing `identity:get_project(project_id=1cc0eb0de66448aa8d924179a50e25e9)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:17.052902 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92180445-2c59-488c-800b-9acaa6825eb6 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:17.053340 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5374c990-5516-47a5-8786-e0e89be0d749 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:17.054853 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 197/391] 10.222.0.27 () {56 vars in 1186 bytes} [Sun Sep 11 00:39:16 2022] GET /identity/v3/projects/1cc0eb0de66448aa8d924179a50e25e9 => generated 298 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:39:17.055771 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 195/392] 10.222.0.27 () {56 vars in 1186 bytes} [Sun Sep 11 00:39:16 2022] GET /identity/v3/projects/1cc0eb0de66448aa8d924179a50e25e9 => generated 298 bytes in 263 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:17.061892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92180445-2c59-488c-800b-9acaa6825eb6 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:17.062041 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5374c990-5516-47a5-8786-e0e89be0d749 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:17.135817 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:17.174686 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5374c990-5516-47a5-8786-e0e89be0d749 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:17.211117 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8aa0a4-ae91-46d2-b4fe-5dfa9972143c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:17.212417 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8aa0a4-ae91-46d2-b4fe-5dfa9972143c None admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:17.212645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8aa0a4-ae91-46d2-b4fe-5dfa9972143c None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:17.212958 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8aa0a4-ae91-46d2-b4fe-5dfa9972143c None admin] PATH_INFO: `/v3/projects/1cc0eb0de66448aa8d924179a50e25e9/groups/3a2cf5af1fa9407ca3402e9d17116e18/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:17.214966 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be8aa0a4-ae91-46d2-b4fe-5dfa9972143c None admin] RBAC: Authorizing `identity:create_grant(project_id=1cc0eb0de66448aa8d924179a50e25e9, group_id=3a2cf5af1fa9407ca3402e9d17116e18, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:17.216350 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be8aa0a4-ae91-46d2-b4fe-5dfa9972143c None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:17.232997 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 196/393] 10.222.0.27 () {58 vars in 1442 bytes} [Sun Sep 11 00:39:17 2022] PUT /identity/v3/projects/1cc0eb0de66448aa8d924179a50e25e9/groups/3a2cf5af1fa9407ca3402e9d17116e18/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:39:17.291830 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92180445-2c59-488c-800b-9acaa6825eb6 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:17.349585 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d436874c-8fd6-4442-b34e-59fb97674100 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:17.350264 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d436874c-8fd6-4442-b34e-59fb97674100 None admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:17.350540 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d436874c-8fd6-4442-b34e-59fb97674100 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:17.350829 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d436874c-8fd6-4442-b34e-59fb97674100 None admin] PATH_INFO: `/v3/projects/1cc0eb0de66448aa8d924179a50e25e9/groups/3a2cf5af1fa9407ca3402e9d17116e18/roles/518eab0f758241369edcdd997b98ad12` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:17.356291 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d436874c-8fd6-4442-b34e-59fb97674100 None admin] RBAC: Authorizing `identity:create_grant(project_id=1cc0eb0de66448aa8d924179a50e25e9, group_id=3a2cf5af1fa9407ca3402e9d17116e18, role_id=518eab0f758241369edcdd997b98ad12)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:17.358630 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d436874c-8fd6-4442-b34e-59fb97674100 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:17.382472 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 198/394] 10.222.0.27 () {58 vars in 1442 bytes} [Sun Sep 11 00:39:17 2022] PUT /identity/v3/projects/1cc0eb0de66448aa8d924179a50e25e9/groups/3a2cf5af1fa9407ca3402e9d17116e18/roles/518eab0f758241369edcdd997b98ad12 => generated 0 bytes in 321 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:39:17.830668 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f0a42f-8794-406e-96d6-33992eb23f55 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:17.831123 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f0a42f-8794-406e-96d6-33992eb23f55 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:17.831538 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f0a42f-8794-406e-96d6-33992eb23f55 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:17.833709 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 197/395] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:39:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:39:17.841780 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2df036-7dbc-4f18-bc9b-888890d4cd24 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:17.842142 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2df036-7dbc-4f18-bc9b-888890d4cd24 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:17.842451 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2df036-7dbc-4f18-bc9b-888890d4cd24 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:17.872060 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-0a2df036-7dbc-4f18-bc9b-888890d4cd24 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:39:17.873277 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-0a2df036-7dbc-4f18-bc9b-888890d4cd24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:39:17.968524 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 199/396] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:39:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 11 00:39:18.151625 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae92989-244b-4882-9cee-aa0e2d6941a3 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:18.151991 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae92989-244b-4882-9cee-aa0e2d6941a3 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:18.152185 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae92989-244b-4882-9cee-aa0e2d6941a3 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:18.153984 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 198/397] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:39:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:39:18.159844 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86f5bdac-62d2-4233-8b50-72cd2f2ebfa8 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:18.160626 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86f5bdac-62d2-4233-8b50-72cd2f2ebfa8 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:18.160836 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86f5bdac-62d2-4233-8b50-72cd2f2ebfa8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:18.189387 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-86f5bdac-62d2-4233-8b50-72cd2f2ebfa8 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:39:18.190584 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-86f5bdac-62d2-4233-8b50-72cd2f2ebfa8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:39:18.213215 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:18.280892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 200/398] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:39:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 11 00:39:18.293941 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a109d5fe-ca03-4828-8e41-38a99d6a8dd6 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:18.294863 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a109d5fe-ca03-4828-8e41-38a99d6a8dd6 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:18.295346 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a109d5fe-ca03-4828-8e41-38a99d6a8dd6 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:18.297105 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 199/399] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:39:18 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:39:18.302289 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce06cc7d-bb05-4414-8a78-510a5d5707b8 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:18.302814 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce06cc7d-bb05-4414-8a78-510a5d5707b8 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:18.303283 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce06cc7d-bb05-4414-8a78-510a5d5707b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:18.323478 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-ce06cc7d-bb05-4414-8a78-510a5d5707b8 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:39:18.324993 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-ce06cc7d-bb05-4414-8a78-510a5d5707b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:39:18.421156 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 201/400] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:39:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:39:18.448038 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a109d5fe-ca03-4828-8e41-38a99d6a8dd6 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:18.474329 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:18.570257 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce06cc7d-bb05-4414-8a78-510a5d5707b8 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:18.597892 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a109d5fe-ca03-4828-8e41-38a99d6a8dd6 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:18.656934 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdfa9459-2799-4985-843d-1b429ee53e64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:18.657670 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdfa9459-2799-4985-843d-1b429ee53e64 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:18.657947 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdfa9459-2799-4985-843d-1b429ee53e64 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:18.658216 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdfa9459-2799-4985-843d-1b429ee53e64 None admin] PATH_INFO: `/v3/roles/member` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:18.673915 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdfa9459-2799-4985-843d-1b429ee53e64 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:39:18.684176 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-bdfa9459-2799-4985-843d-1b429ee53e64 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 11 00:39:18.684176 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:18.684176 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:18.684176 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:18.684176 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:18.684176 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:18.684176 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:18.684176 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:18.684176 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:18.684176 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:18.684176 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:18.684176 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:18.684176 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:39:18.684176 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:39:18.684176 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:39:18.684176 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:39:18.685072 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:39:18.685072 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:39:18.685072 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:39:18.685072 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:39:18.685072 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:18.685072 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:18.685072 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:18.685072 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:18.685072 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:18.685072 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:18.685072 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:18.685072 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:18.685072 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:18.685072 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:18.685072 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:18.685072 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:18.686327 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:18.686327 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:18.686327 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:18.686327 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:18.686327 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:39:18.686327 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:39:18.686327 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:39:18.686327 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:39:18.686327 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:39:18.686327 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:39:18.686327 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 11 00:39:18.686327 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:18.686984 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 200/401] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:39:18 2022] GET /identity/v3/roles/member => generated 84 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:18.692988 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdfa9459-2799-4985-843d-1b429ee53e64 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:18.739149 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:18.773063 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce06cc7d-bb05-4414-8a78-510a5d5707b8 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:18.800469 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:18.813564 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e39257c7-a227-4573-89a4-262eb331ff0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:18.814194 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39257c7-a227-4573-89a4-262eb331ff0f None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:18.814466 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39257c7-a227-4573-89a4-262eb331ff0f None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:18.814731 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39257c7-a227-4573-89a4-262eb331ff0f None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:18.828193 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdfa9459-2799-4985-843d-1b429ee53e64 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:18.830120 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e39257c7-a227-4573-89a4-262eb331ff0f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:39:18.835970 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-e39257c7-a227-4573-89a4-262eb331ff0f None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 11 00:39:18.835970 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:18.835970 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:18.835970 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:18.835970 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:18.835970 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:18.835970 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:18.835970 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:18.835970 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:18.835970 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:18.835970 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:18.835970 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:18.835970 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:39:18.835970 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:39:18.835970 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:39:18.835970 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:39:18.837420 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:39:18.837420 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:39:18.837420 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:39:18.837420 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:39:18.837420 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:18.837420 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:18.837420 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:18.837420 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:18.837420 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:18.837420 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:18.837420 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:18.837420 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:18.837420 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:18.837420 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:18.837420 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:18.837420 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:18.838218 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:18.838218 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:18.838218 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:18.838218 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:18.838218 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:39:18.838218 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:39:18.838218 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:39:18.838218 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:39:18.838218 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:39:18.838218 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:39:18.838218 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 11 00:39:18.838218 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:18.839171 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 202/402] 10.222.0.27 () {56 vars in 1114 bytes} [Sun Sep 11 00:39:18 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 269 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:18.840220 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e39257c7-a227-4573-89a4-262eb331ff0f None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:18.870542 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09227d27-b485-4dd9-873a-12666495742e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:18.871420 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09227d27-b485-4dd9-873a-12666495742e None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:18.871792 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09227d27-b485-4dd9-873a-12666495742e None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:18.872207 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09227d27-b485-4dd9-873a-12666495742e None admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:18.872981 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09227d27-b485-4dd9-873a-12666495742e None admin] RBAC: Adding query filter params (name=member) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:18.873378 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09227d27-b485-4dd9-873a-12666495742e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:18.875054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09227d27-b485-4dd9-873a-12666495742e None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:18.885322 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 201/403] 10.222.0.27 () {56 vars in 1101 bytes} [Sun Sep 11 00:39:18 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:18.886329 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09227d27-b485-4dd9-873a-12666495742e None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:18.955644 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e39257c7-a227-4573-89a4-262eb331ff0f None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:18.995752 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20954045-9975-4b71-9dee-6f14c06984da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:18.996393 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20954045-9975-4b71-9dee-6f14c06984da None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:18.996673 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20954045-9975-4b71-9dee-6f14c06984da None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:18.996949 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20954045-9975-4b71-9dee-6f14c06984da None admin] PATH_INFO: `/v3/roles/admin` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:18.997983 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09227d27-b485-4dd9-873a-12666495742e None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:19.005187 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:19.011102 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20954045-9975-4b71-9dee-6f14c06984da None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:39:19.017260 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-20954045-9975-4b71-9dee-6f14c06984da None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:39:19.017260 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:19.017260 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:19.017260 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:19.017260 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:19.017260 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:19.017260 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:19.017260 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:19.017260 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:19.017260 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:19.017260 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:19.017260 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:19.017260 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:39:19.017260 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:39:19.017260 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:39:19.017260 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:39:19.018226 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:39:19.018226 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:39:19.018226 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:39:19.018226 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:39:19.018226 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:19.018226 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:19.018226 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:19.018226 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:19.018226 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:19.018226 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:19.018226 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:19.018226 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:19.018226 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:19.018226 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:19.018226 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:19.018226 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:19.019058 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:19.019058 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:19.019058 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:19.019058 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:19.019058 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:39:19.019058 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:39:19.019058 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:39:19.019058 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:39:19.019058 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:39:19.019058 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:39:19.019058 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:39:19.019058 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:19.020481 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 203/404] 10.222.0.27 () {56 vars in 1096 bytes} [Sun Sep 11 00:39:18 2022] GET /identity/v3/roles/admin => generated 83 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:19.022134 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20954045-9975-4b71-9dee-6f14c06984da None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:19.043593 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff398f51-6a62-41ba-b831-f4f0364313b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:19.044205 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff398f51-6a62-41ba-b831-f4f0364313b2 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:19.044480 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff398f51-6a62-41ba-b831-f4f0364313b2 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:19.044734 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff398f51-6a62-41ba-b831-f4f0364313b2 None admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:19.045391 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff398f51-6a62-41ba-b831-f4f0364313b2 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:19.045692 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff398f51-6a62-41ba-b831-f4f0364313b2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:19.047197 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff398f51-6a62-41ba-b831-f4f0364313b2 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:19.049631 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:19.059221 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 202/405] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:39:18 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:19.060248 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff398f51-6a62-41ba-b831-f4f0364313b2 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:19.139302 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20954045-9975-4b71-9dee-6f14c06984da None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:19.179194 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74cb70ee-8e79-4856-b307-82ee265bdde8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:19.179875 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74cb70ee-8e79-4856-b307-82ee265bdde8 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:19.181218 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74cb70ee-8e79-4856-b307-82ee265bdde8 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:19.181747 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74cb70ee-8e79-4856-b307-82ee265bdde8 None admin] PATH_INFO: `/v3/groups/3a2cf5af1fa9407ca3402e9d17116e18` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:19.183313 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74cb70ee-8e79-4856-b307-82ee265bdde8 None admin] RBAC: Authorizing `identity:get_group(group_id=3a2cf5af1fa9407ca3402e9d17116e18)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:19.185056 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74cb70ee-8e79-4856-b307-82ee265bdde8 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:19.185780 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff398f51-6a62-41ba-b831-f4f0364313b2 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:19.186677 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 204/406] 10.222.0.27 () {56 vars in 1180 bytes} [Sun Sep 11 00:39:19 2022] GET /identity/v3/groups/3a2cf5af1fa9407ca3402e9d17116e18 => generated 226 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:19.187558 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74cb70ee-8e79-4856-b307-82ee265bdde8 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:19.230874 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd5cb729-8cd8-4e0f-a90d-566b29e6bfa9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:19.231412 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5cb729-8cd8-4e0f-a90d-566b29e6bfa9 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:19.231684 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5cb729-8cd8-4e0f-a90d-566b29e6bfa9 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:19.231949 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5cb729-8cd8-4e0f-a90d-566b29e6bfa9 None admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:19.232606 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd5cb729-8cd8-4e0f-a90d-566b29e6bfa9 None admin] RBAC: Adding query filter params (name=admin) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:19.232901 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd5cb729-8cd8-4e0f-a90d-566b29e6bfa9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:19.234380 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd5cb729-8cd8-4e0f-a90d-566b29e6bfa9 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:19.244384 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 203/407] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:39:19 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:19.249636 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd5cb729-8cd8-4e0f-a90d-566b29e6bfa9 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:19.263434 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:19.306964 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74cb70ee-8e79-4856-b307-82ee265bdde8 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:19.322103 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:19.347853 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06858581-a912-401e-9592-009544402203 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:19.348810 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06858581-a912-401e-9592-009544402203 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:19.349272 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06858581-a912-401e-9592-009544402203 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:19.349733 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06858581-a912-401e-9592-009544402203 None admin] PATH_INFO: `/v3/groups/3a2cf5af1fa9407ca3402e9d17116e18` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:19.351121 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06858581-a912-401e-9592-009544402203 None admin] RBAC: Authorizing `identity:get_group(group_id=3a2cf5af1fa9407ca3402e9d17116e18)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:19.353757 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06858581-a912-401e-9592-009544402203 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:19.355588 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 205/408] 10.222.0.27 () {56 vars in 1180 bytes} [Sun Sep 11 00:39:19 2022] GET /identity/v3/groups/3a2cf5af1fa9407ca3402e9d17116e18 => generated 226 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:19.356953 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06858581-a912-401e-9592-009544402203 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:19.370140 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd5cb729-8cd8-4e0f-a90d-566b29e6bfa9 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:19.410690 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf3ffbe-c7c5-484c-8ee9-4346c777ddf4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:19.411457 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf3ffbe-c7c5-484c-8ee9-4346c777ddf4 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:19.411741 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf3ffbe-c7c5-484c-8ee9-4346c777ddf4 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:19.412017 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf3ffbe-c7c5-484c-8ee9-4346c777ddf4 None admin] PATH_INFO: `/v3/projects/a3af44366c3a450994b868f0b5428c43` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:19.413155 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cf3ffbe-c7c5-484c-8ee9-4346c777ddf4 None admin] RBAC: Authorizing `identity:get_project(project_id=a3af44366c3a450994b868f0b5428c43)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:19.415290 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cf3ffbe-c7c5-484c-8ee9-4346c777ddf4 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:19.417360 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 204/409] 10.222.0.27 () {56 vars in 1186 bytes} [Sun Sep 11 00:39:19 2022] GET /identity/v3/projects/a3af44366c3a450994b868f0b5428c43 => generated 302 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:19.418237 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf3ffbe-c7c5-484c-8ee9-4346c777ddf4 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:19.481096 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06858581-a912-401e-9592-009544402203 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:19.519265 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21f4be9e-7420-451f-b565-6b66307809ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:19.520799 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21f4be9e-7420-451f-b565-6b66307809ce None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:19.521279 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21f4be9e-7420-451f-b565-6b66307809ce None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:19.521768 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21f4be9e-7420-451f-b565-6b66307809ce None admin] PATH_INFO: `/v3/groups/dbc01bc745ca46db93dee6ac3ec2c72e` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:19.523173 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21f4be9e-7420-451f-b565-6b66307809ce None admin] RBAC: Authorizing `identity:get_group(group_id=dbc01bc745ca46db93dee6ac3ec2c72e)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:19.525385 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21f4be9e-7420-451f-b565-6b66307809ce None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:19.527924 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 206/410] 10.222.0.27 () {56 vars in 1180 bytes} [Sun Sep 11 00:39:19 2022] GET /identity/v3/groups/dbc01bc745ca46db93dee6ac3ec2c72e => generated 229 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:19.528864 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21f4be9e-7420-451f-b565-6b66307809ce None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:19.529262 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf3ffbe-c7c5-484c-8ee9-4346c777ddf4 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:19.538408 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:19.564682 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:19.574308 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f29689d0-f616-456e-9434-cc5d62269673 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:19.574973 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29689d0-f616-456e-9434-cc5d62269673 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:19.575260 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29689d0-f616-456e-9434-cc5d62269673 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:19.575538 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29689d0-f616-456e-9434-cc5d62269673 None admin] PATH_INFO: `/v3/projects/a3af44366c3a450994b868f0b5428c43` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:19.576777 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f29689d0-f616-456e-9434-cc5d62269673 None admin] RBAC: Authorizing `identity:get_project(project_id=a3af44366c3a450994b868f0b5428c43)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:19.578460 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f29689d0-f616-456e-9434-cc5d62269673 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:19.580441 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 205/411] 10.222.0.27 () {56 vars in 1186 bytes} [Sun Sep 11 00:39:19 2022] GET /identity/v3/projects/a3af44366c3a450994b868f0b5428c43 => generated 302 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:19.581290 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f29689d0-f616-456e-9434-cc5d62269673 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:19.645658 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21f4be9e-7420-451f-b565-6b66307809ce None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:19.690815 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed971d5f-4c0e-4e1b-be70-2e6014964d03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:19.691498 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed971d5f-4c0e-4e1b-be70-2e6014964d03 None admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:19.691834 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed971d5f-4c0e-4e1b-be70-2e6014964d03 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:19.692116 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed971d5f-4c0e-4e1b-be70-2e6014964d03 None admin] PATH_INFO: `/v3/projects/a3af44366c3a450994b868f0b5428c43/groups/3a2cf5af1fa9407ca3402e9d17116e18/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:19.694226 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed971d5f-4c0e-4e1b-be70-2e6014964d03 None admin] RBAC: Authorizing `identity:create_grant(project_id=a3af44366c3a450994b868f0b5428c43, group_id=3a2cf5af1fa9407ca3402e9d17116e18, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:19.696322 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed971d5f-4c0e-4e1b-be70-2e6014964d03 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:19.709823 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f29689d0-f616-456e-9434-cc5d62269673 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:19.714766 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 207/412] 10.222.0.27 () {58 vars in 1442 bytes} [Sun Sep 11 00:39:19 2022] PUT /identity/v3/projects/a3af44366c3a450994b868f0b5428c43/groups/3a2cf5af1fa9407ca3402e9d17116e18/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:39:19.715658 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed971d5f-4c0e-4e1b-be70-2e6014964d03 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:19.772609 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12504ff1-f79e-4b04-8e6d-51549f2a253f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:19.773623 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12504ff1-f79e-4b04-8e6d-51549f2a253f None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:19.774215 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12504ff1-f79e-4b04-8e6d-51549f2a253f None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:19.774710 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12504ff1-f79e-4b04-8e6d-51549f2a253f None admin] PATH_INFO: `/v3/projects/12356f1769944cc8be5c533d062adf65` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:19.777393 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12504ff1-f79e-4b04-8e6d-51549f2a253f None admin] RBAC: Authorizing `identity:get_project(project_id=12356f1769944cc8be5c533d062adf65)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:19.780609 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12504ff1-f79e-4b04-8e6d-51549f2a253f None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:19.787146 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 206/413] 10.222.0.27 () {56 vars in 1186 bytes} [Sun Sep 11 00:39:19 2022] GET /identity/v3/projects/12356f1769944cc8be5c533d062adf65 => generated 344 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:19.794723 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12504ff1-f79e-4b04-8e6d-51549f2a253f None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:19.861774 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:19.929887 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed971d5f-4c0e-4e1b-be70-2e6014964d03 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:19.934258 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:19.972164 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baf5b9c3-23d9-4b7a-88f3-c399125ea035 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:19.973011 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf5b9c3-23d9-4b7a-88f3-c399125ea035 None admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:19.973609 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf5b9c3-23d9-4b7a-88f3-c399125ea035 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:19.973903 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf5b9c3-23d9-4b7a-88f3-c399125ea035 None admin] PATH_INFO: `/v3/projects/a3af44366c3a450994b868f0b5428c43/groups/3a2cf5af1fa9407ca3402e9d17116e18/roles/518eab0f758241369edcdd997b98ad12` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:19.977318 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baf5b9c3-23d9-4b7a-88f3-c399125ea035 None admin] RBAC: Authorizing `identity:create_grant(project_id=a3af44366c3a450994b868f0b5428c43, group_id=3a2cf5af1fa9407ca3402e9d17116e18, role_id=518eab0f758241369edcdd997b98ad12)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:19.979132 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baf5b9c3-23d9-4b7a-88f3-c399125ea035 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:19.982507 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12504ff1-f79e-4b04-8e6d-51549f2a253f None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:19.993788 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 208/414] 10.222.0.27 () {58 vars in 1442 bytes} [Sun Sep 11 00:39:19 2022] PUT /identity/v3/projects/a3af44366c3a450994b868f0b5428c43/groups/3a2cf5af1fa9407ca3402e9d17116e18/roles/518eab0f758241369edcdd997b98ad12 => generated 0 bytes in 278 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:39:19.998834 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d38a06-0e72-42d1-8737-b248efc576ac None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:19.998834 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d38a06-0e72-42d1-8737-b248efc576ac None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:19.998834 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d38a06-0e72-42d1-8737-b248efc576ac None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:19.998834 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 209/415] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:39:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:39:19.998834 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfae1dc-0d96-48be-9fb7-cff45cb3b102 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:20.003153 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfae1dc-0d96-48be-9fb7-cff45cb3b102 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:20.003497 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfae1dc-0d96-48be-9fb7-cff45cb3b102 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:20.005855 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 210/416] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:39:19 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:39:20.007082 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa69f576-d729-4a97-9983-7974607cfc47 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:20.007365 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa69f576-d729-4a97-9983-7974607cfc47 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:20.007634 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa69f576-d729-4a97-9983-7974607cfc47 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:20.026376 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd313c3-f68a-443c-8397-b68eba5b09e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:20.027317 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd313c3-f68a-443c-8397-b68eba5b09e7 None admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:20.027736 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd313c3-f68a-443c-8397-b68eba5b09e7 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:20.028145 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd313c3-f68a-443c-8397-b68eba5b09e7 None admin] PATH_INFO: `/v3/projects/12356f1769944cc8be5c533d062adf65/groups/dbc01bc745ca46db93dee6ac3ec2c72e/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:20.029995 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-fa69f576-d729-4a97-9983-7974607cfc47 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:39:20.030557 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dd313c3-f68a-443c-8397-b68eba5b09e7 None admin] RBAC: Authorizing `identity:create_grant(project_id=12356f1769944cc8be5c533d062adf65, group_id=dbc01bc745ca46db93dee6ac3ec2c72e, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:20.031413 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-fa69f576-d729-4a97-9983-7974607cfc47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:39:20.032427 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dd313c3-f68a-443c-8397-b68eba5b09e7 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:20.048345 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 207/417] 10.222.0.27 () {58 vars in 1442 bytes} [Sun Sep 11 00:39:19 2022] PUT /identity/v3/projects/12356f1769944cc8be5c533d062adf65/groups/dbc01bc745ca46db93dee6ac3ec2c72e/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 255 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:39:20.049923 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8403e305-01c4-48bb-9cc8-e5232105654f None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:20.050371 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8403e305-01c4-48bb-9cc8-e5232105654f None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:20.050800 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8403e305-01c4-48bb-9cc8-e5232105654f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:20.075414 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-8403e305-01c4-48bb-9cc8-e5232105654f None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:39:20.076908 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-8403e305-01c4-48bb-9cc8-e5232105654f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:39:20.161322 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 208/418] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:39:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:39:20.170108 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 211/419] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:39:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 11 00:39:20.447402 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8403e305-01c4-48bb-9cc8-e5232105654f None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:20.507223 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:20.564162 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8403e305-01c4-48bb-9cc8-e5232105654f None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:20.571944 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa69f576-d729-4a97-9983-7974607cfc47 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:20.623518 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ca95c4-0be0-4575-ad16-3499c8a26605 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:20.624161 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ca95c4-0be0-4575-ad16-3499c8a26605 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:20.624438 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ca95c4-0be0-4575-ad16-3499c8a26605 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:20.624706 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ca95c4-0be0-4575-ad16-3499c8a26605 None admin] PATH_INFO: `/v3/roles/member` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:20.648465 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2ca95c4-0be0-4575-ad16-3499c8a26605 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:39:20.655348 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-a2ca95c4-0be0-4575-ad16-3499c8a26605 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 11 00:39:20.655348 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:20.655348 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:20.655348 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:20.655348 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:20.655348 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:20.655348 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:20.655348 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:20.655348 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:20.655348 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:20.655348 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:20.655348 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:20.655348 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:39:20.655348 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:39:20.655348 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:39:20.655348 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:39:20.656226 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:39:20.656226 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:39:20.656226 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:39:20.656226 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:39:20.656226 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:20.656226 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:20.656226 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:20.656226 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:20.656226 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:20.656226 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:20.656226 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:20.656226 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:20.656226 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:20.656226 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:20.656226 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:20.656226 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:20.657041 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:20.657041 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:20.657041 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:20.657041 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:20.657041 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:39:20.657041 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:39:20.657041 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:39:20.657041 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:39:20.657041 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:39:20.657041 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:39:20.657041 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 11 00:39:20.657041 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:20.657041 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 209/420] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:39:20 2022] GET /identity/v3/roles/member => generated 84 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:20.664010 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ca95c4-0be0-4575-ad16-3499c8a26605 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:20.671185 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:20.757095 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa69f576-d729-4a97-9983-7974607cfc47 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:20.786889 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ca95c4-0be0-4575-ad16-3499c8a26605 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:20.803748 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:20.830894 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ef81b1a-036d-458a-8fd1-711c7c640167 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:20.831744 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef81b1a-036d-458a-8fd1-711c7c640167 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:20.832129 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef81b1a-036d-458a-8fd1-711c7c640167 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:20.832411 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef81b1a-036d-458a-8fd1-711c7c640167 None admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:20.833085 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ef81b1a-036d-458a-8fd1-711c7c640167 None admin] RBAC: Adding query filter params (name=member) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:20.833391 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ef81b1a-036d-458a-8fd1-711c7c640167 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:20.834767 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ef81b1a-036d-458a-8fd1-711c7c640167 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:20.840552 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d60193bf-570c-4480-858a-df0f007de5df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:20.841209 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d60193bf-570c-4480-858a-df0f007de5df None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:20.841484 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d60193bf-570c-4480-858a-df0f007de5df None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:20.841750 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d60193bf-570c-4480-858a-df0f007de5df None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:20.844984 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 210/421] 10.222.0.27 () {56 vars in 1101 bytes} [Sun Sep 11 00:39:20 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:20.850710 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ef81b1a-036d-458a-8fd1-711c7c640167 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:20.871283 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d60193bf-570c-4480-858a-df0f007de5df None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:39:20.876198 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-d60193bf-570c-4480-858a-df0f007de5df None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 11 00:39:20.876198 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:20.876198 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:20.876198 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:20.876198 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:20.876198 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:20.876198 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:20.876198 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:20.876198 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:20.876198 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:20.876198 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:20.876198 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:20.876198 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:39:20.876198 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:39:20.876198 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:39:20.876198 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:39:20.877088 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:39:20.877088 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:39:20.877088 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:39:20.877088 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:39:20.877088 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:20.877088 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:20.877088 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:20.877088 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:20.877088 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:20.877088 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:20.877088 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:20.877088 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:20.877088 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:20.877088 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:20.877088 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:20.877088 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:20.877890 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:20.877890 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:20.877890 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:20.877890 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:20.877890 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:39:20.877890 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:39:20.877890 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:39:20.877890 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:39:20.877890 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:39:20.877890 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:39:20.877890 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 11 00:39:20.877890 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:20.877890 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 212/422] 10.222.0.27 () {56 vars in 1114 bytes} [Sun Sep 11 00:39:20 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 307 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:20.885527 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d60193bf-570c-4480-858a-df0f007de5df None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:21.014597 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ef81b1a-036d-458a-8fd1-711c7c640167 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:21.047440 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d60193bf-570c-4480-858a-df0f007de5df None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:21.062528 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:21.095136 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-364fef73-56ff-434e-bf46-4bd0830323a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:21.097187 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-364fef73-56ff-434e-bf46-4bd0830323a9 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:21.098223 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-364fef73-56ff-434e-bf46-4bd0830323a9 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:21.099240 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-364fef73-56ff-434e-bf46-4bd0830323a9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:21.102772 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f28e032-2b16-4a3b-a818-d72295e8fedc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:21.110811 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f28e032-2b16-4a3b-a818-d72295e8fedc None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:21.110811 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f28e032-2b16-4a3b-a818-d72295e8fedc None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:21.110811 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f28e032-2b16-4a3b-a818-d72295e8fedc None admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:21.110811 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f28e032-2b16-4a3b-a818-d72295e8fedc None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:21.110811 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f28e032-2b16-4a3b-a818-d72295e8fedc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:21.110811 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f28e032-2b16-4a3b-a818-d72295e8fedc None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:21.130958 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 213/423] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:39:20 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:21.133748 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f28e032-2b16-4a3b-a818-d72295e8fedc None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:21.205364 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:21.249376 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-364fef73-56ff-434e-bf46-4bd0830323a9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:21.250871 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-364fef73-56ff-434e-bf46-4bd0830323a9 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:21.324028 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f28e032-2b16-4a3b-a818-d72295e8fedc None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:21.360890 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce4830a-6f73-4d0a-91a0-c0aedfabc956 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:21.361496 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce4830a-6f73-4d0a-91a0-c0aedfabc956 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:21.361776 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce4830a-6f73-4d0a-91a0-c0aedfabc956 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:21.365950 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce4830a-6f73-4d0a-91a0-c0aedfabc956 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:21.439836 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:21.448858 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 211/424] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:39:20 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 598 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 11 00:39:21.456361 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-364fef73-56ff-434e-bf46-4bd0830323a9 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:21.494430 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cce4830a-6f73-4d0a-91a0-c0aedfabc956 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:21.496196 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cce4830a-6f73-4d0a-91a0-c0aedfabc956 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:21.560695 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:21.609034 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-364fef73-56ff-434e-bf46-4bd0830323a9 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:21.621711 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 214/425] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:39:21 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 489 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 11 00:39:21.627745 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce4830a-6f73-4d0a-91a0-c0aedfabc956 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:21.665058 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5bd4f8c-7589-4290-af54-c9223ed496cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:21.666011 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5bd4f8c-7589-4290-af54-c9223ed496cb None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:21.666289 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5bd4f8c-7589-4290-af54-c9223ed496cb None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:21.666555 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5bd4f8c-7589-4290-af54-c9223ed496cb None admin] PATH_INFO: `/v3/projects/1cc0eb0de66448aa8d924179a50e25e9` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:21.668674 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5bd4f8c-7589-4290-af54-c9223ed496cb None admin] RBAC: Authorizing `identity:get_project(project_id=1cc0eb0de66448aa8d924179a50e25e9)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:21.670176 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5bd4f8c-7589-4290-af54-c9223ed496cb None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:21.671843 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 212/426] 10.222.0.27 () {56 vars in 1186 bytes} [Sun Sep 11 00:39:21 2022] GET /identity/v3/projects/1cc0eb0de66448aa8d924179a50e25e9 => generated 298 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:21.677365 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5bd4f8c-7589-4290-af54-c9223ed496cb None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:21.761736 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:21.808927 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce4830a-6f73-4d0a-91a0-c0aedfabc956 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:21.856509 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5bd4f8c-7589-4290-af54-c9223ed496cb None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:21.872027 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c1ad45-ab9e-40a7-b6ac-36e2f402492c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:21.872590 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c1ad45-ab9e-40a7-b6ac-36e2f402492c None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:21.872865 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c1ad45-ab9e-40a7-b6ac-36e2f402492c None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:21.873126 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c1ad45-ab9e-40a7-b6ac-36e2f402492c None admin] PATH_INFO: `/v3/projects/1cc0eb0de66448aa8d924179a50e25e9` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:21.874534 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95c1ad45-ab9e-40a7-b6ac-36e2f402492c None admin] RBAC: Authorizing `identity:get_project(project_id=1cc0eb0de66448aa8d924179a50e25e9)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:21.878853 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95c1ad45-ab9e-40a7-b6ac-36e2f402492c None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:21.881539 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 215/427] 10.222.0.27 () {56 vars in 1186 bytes} [Sun Sep 11 00:39:21 2022] GET /identity/v3/projects/1cc0eb0de66448aa8d924179a50e25e9 => generated 298 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:39:21.887322 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c1ad45-ab9e-40a7-b6ac-36e2f402492c None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:21.944816 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d8ea4a-6473-4138-b084-a50ea2f9e368 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:21.954184 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d8ea4a-6473-4138-b084-a50ea2f9e368 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:21.954500 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d8ea4a-6473-4138-b084-a50ea2f9e368 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:21.954825 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d8ea4a-6473-4138-b084-a50ea2f9e368 None admin] PATH_INFO: `/v3/groups/3a2cf5af1fa9407ca3402e9d17116e18` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:21.957587 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75d8ea4a-6473-4138-b084-a50ea2f9e368 None admin] RBAC: Authorizing `identity:get_group(group_id=3a2cf5af1fa9407ca3402e9d17116e18)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:21.959080 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75d8ea4a-6473-4138-b084-a50ea2f9e368 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:21.961033 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 213/428] 10.222.0.27 () {56 vars in 1180 bytes} [Sun Sep 11 00:39:21 2022] GET /identity/v3/groups/3a2cf5af1fa9407ca3402e9d17116e18 => generated 226 bytes in 284 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:39:21.966902 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d8ea4a-6473-4138-b084-a50ea2f9e368 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:22.003292 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:22.006582 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c1ad45-ab9e-40a7-b6ac-36e2f402492c None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:22.062860 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b22489f-c911-45e9-9edb-b558fd83dd8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:22.063919 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b22489f-c911-45e9-9edb-b558fd83dd8e None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:22.064417 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b22489f-c911-45e9-9edb-b558fd83dd8e None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:22.064904 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b22489f-c911-45e9-9edb-b558fd83dd8e None admin] PATH_INFO: `/v3/groups/3a2cf5af1fa9407ca3402e9d17116e18` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:22.068329 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b22489f-c911-45e9-9edb-b558fd83dd8e None admin] RBAC: Authorizing `identity:get_group(group_id=3a2cf5af1fa9407ca3402e9d17116e18)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:22.070026 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b22489f-c911-45e9-9edb-b558fd83dd8e None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:22.072433 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 216/429] 10.222.0.27 () {56 vars in 1180 bytes} [Sun Sep 11 00:39:21 2022] GET /identity/v3/groups/3a2cf5af1fa9407ca3402e9d17116e18 => generated 226 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:22.078406 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b22489f-c911-45e9-9edb-b558fd83dd8e None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:22.109497 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:22.209107 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d8ea4a-6473-4138-b084-a50ea2f9e368 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:22.221842 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b22489f-c911-45e9-9edb-b558fd83dd8e None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:22.259643 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e8466a7-a187-4d87-815e-4f059ca24299 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:22.260253 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8466a7-a187-4d87-815e-4f059ca24299 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:22.260645 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8466a7-a187-4d87-815e-4f059ca24299 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:22.260920 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8466a7-a187-4d87-815e-4f059ca24299 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:22.263311 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e8466a7-a187-4d87-815e-4f059ca24299 None admin] RBAC: Adding query filter params (group.id=3a2cf5af1fa9407ca3402e9d17116e18, role.id=518eab0f758241369edcdd997b98ad12, scope.project.id=1cc0eb0de66448aa8d924179a50e25e9) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:22.263311 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e8466a7-a187-4d87-815e-4f059ca24299 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:22.263311 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e8466a7-a187-4d87-815e-4f059ca24299 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:22.273280 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 217/430] 10.222.0.27 () {56 vars in 1376 bytes} [Sun Sep 11 00:39:22 2022] GET /identity/v3/role_assignments?group.id=3a2cf5af1fa9407ca3402e9d17116e18&scope.project.id=1cc0eb0de66448aa8d924179a50e25e9&role.id=518eab0f758241369edcdd997b98ad12 => generated 611 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:22.294652 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5d82aa2-c3d2-433b-b5b3-53b9bc8519ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:22.295005 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a075563-353b-4869-915a-fd931cbb6922 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:22.295005 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a075563-353b-4869-915a-fd931cbb6922 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:22.295335 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a075563-353b-4869-915a-fd931cbb6922 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:22.295487 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d82aa2-c3d2-433b-b5b3-53b9bc8519ab None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:22.295680 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d82aa2-c3d2-433b-b5b3-53b9bc8519ab None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:22.295825 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d82aa2-c3d2-433b-b5b3-53b9bc8519ab None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:22.296544 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5d82aa2-c3d2-433b-b5b3-53b9bc8519ab None admin] RBAC: Adding query filter params (group.id=3a2cf5af1fa9407ca3402e9d17116e18, role.id=2e64fc597b6447df99a7992d411d1a34, scope.project.id=1cc0eb0de66448aa8d924179a50e25e9) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:22.296852 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5d82aa2-c3d2-433b-b5b3-53b9bc8519ab None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:22.297321 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 218/431] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:39:22 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 11 00:39:22.298438 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5d82aa2-c3d2-433b-b5b3-53b9bc8519ab None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:22.303124 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4719548-b452-4153-8433-8984bf8e8d73 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:22.303421 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4719548-b452-4153-8433-8984bf8e8d73 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:22.303666 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4719548-b452-4153-8433-8984bf8e8d73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:22.314944 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 214/432] 10.222.0.27 () {56 vars in 1376 bytes} [Sun Sep 11 00:39:21 2022] GET /identity/v3/role_assignments?group.id=3a2cf5af1fa9407ca3402e9d17116e18&scope.project.id=1cc0eb0de66448aa8d924179a50e25e9&role.id=2e64fc597b6447df99a7992d411d1a34 => generated 611 bytes in 343 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:39:22.324685 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-e4719548-b452-4153-8433-8984bf8e8d73 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:39:22.325932 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-e4719548-b452-4153-8433-8984bf8e8d73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:39:22.403214 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 219/433] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:39:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:39:22.685551 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5d82aa2-c3d2-433b-b5b3-53b9bc8519ab None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:22.722244 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df983c1-fe67-4d43-a4be-5e40b4f24274 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:22.722902 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df983c1-fe67-4d43-a4be-5e40b4f24274 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:22.722902 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df983c1-fe67-4d43-a4be-5e40b4f24274 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:22.724707 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 220/434] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:39:22 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:39:22.730949 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f647546-0412-431b-a7d1-271fcdbdacc7 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:22.731232 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f647546-0412-431b-a7d1-271fcdbdacc7 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:22.732369 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f647546-0412-431b-a7d1-271fcdbdacc7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:22.741134 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:22.755093 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-3f647546-0412-431b-a7d1-271fcdbdacc7 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:39:22.756295 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-3f647546-0412-431b-a7d1-271fcdbdacc7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:39:22.782262 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:22.809579 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5d82aa2-c3d2-433b-b5b3-53b9bc8519ab None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:22.833314 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 221/435] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:39:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 11 00:39:22.834672 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70d770d-4e19-4500-8b07-f7a6dbc7f276 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:22.834999 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70d770d-4e19-4500-8b07-f7a6dbc7f276 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:22.835276 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70d770d-4e19-4500-8b07-f7a6dbc7f276 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:22.837697 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 222/436] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:39:22 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:39:22.842482 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b99eda9a-18f6-4d19-863e-99d9b31d086f None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:22.842767 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b99eda9a-18f6-4d19-863e-99d9b31d086f None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:22.843108 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b99eda9a-18f6-4d19-863e-99d9b31d086f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:22.848890 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c20663da-991c-45ca-816d-ea6c818bc2e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:22.849519 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20663da-991c-45ca-816d-ea6c818bc2e5 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:22.849797 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20663da-991c-45ca-816d-ea6c818bc2e5 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:22.850061 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20663da-991c-45ca-816d-ea6c818bc2e5 None admin] PATH_INFO: `/v3/roles/member` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:22.863087 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c20663da-991c-45ca-816d-ea6c818bc2e5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:39:22.865293 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-b99eda9a-18f6-4d19-863e-99d9b31d086f None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:39:22.866683 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-b99eda9a-18f6-4d19-863e-99d9b31d086f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:39:22.870879 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-c20663da-991c-45ca-816d-ea6c818bc2e5 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 11 00:39:22.870879 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:22.870879 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:22.870879 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:22.870879 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:22.870879 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:22.870879 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:22.870879 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:22.870879 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:22.870879 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:22.870879 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:22.870879 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:22.870879 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:39:22.870879 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:39:22.870879 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:39:22.870879 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:39:22.871811 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:39:22.871811 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:39:22.871811 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:39:22.871811 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:39:22.871811 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:22.871811 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:22.871811 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:22.871811 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:22.871811 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:22.871811 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:22.871811 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:22.871811 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:22.871811 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:22.871811 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:22.871811 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:22.871811 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:22.872630 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:22.872630 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:22.872630 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:22.872630 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:22.872630 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:39:22.872630 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:39:22.872630 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:39:22.872630 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:39:22.872630 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:39:22.872630 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:39:22.872630 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 11 00:39:22.872630 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:22.875882 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 215/437] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:39:22 2022] GET /identity/v3/roles/member => generated 84 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:22.877980 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c20663da-991c-45ca-816d-ea6c818bc2e5 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:22.949810 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 223/438] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:39:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 11 00:39:22.996006 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c20663da-991c-45ca-816d-ea6c818bc2e5 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:23.033247 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de24e534-e697-4075-9235-b9c79d04d655 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:23.034193 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de24e534-e697-4075-9235-b9c79d04d655 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:23.034661 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de24e534-e697-4075-9235-b9c79d04d655 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:23.035165 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de24e534-e697-4075-9235-b9c79d04d655 None admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:23.036319 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de24e534-e697-4075-9235-b9c79d04d655 None admin] RBAC: Adding query filter params (name=member) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:23.036809 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de24e534-e697-4075-9235-b9c79d04d655 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:23.038366 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de24e534-e697-4075-9235-b9c79d04d655 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:23.041278 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:23.052347 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 216/439] 10.222.0.27 () {56 vars in 1101 bytes} [Sun Sep 11 00:39:22 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:23.055940 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b99eda9a-18f6-4d19-863e-99d9b31d086f None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:23.116528 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de24e534-e697-4075-9235-b9c79d04d655 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:23.131297 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:23.173880 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b99eda9a-18f6-4d19-863e-99d9b31d086f None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:23.213680 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f90f244-fd9e-4fbc-b144-ab6cb568ac23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:23.214287 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f90f244-fd9e-4fbc-b144-ab6cb568ac23 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:23.214571 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f90f244-fd9e-4fbc-b144-ab6cb568ac23 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:23.214865 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f90f244-fd9e-4fbc-b144-ab6cb568ac23 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:23.227218 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de24e534-e697-4075-9235-b9c79d04d655 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:23.261226 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e937d54-3d41-4164-a304-cfd5284aba47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:23.261853 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e937d54-3d41-4164-a304-cfd5284aba47 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:23.262130 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e937d54-3d41-4164-a304-cfd5284aba47 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:23.262402 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e937d54-3d41-4164-a304-cfd5284aba47 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:23.272498 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e937d54-3d41-4164-a304-cfd5284aba47 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:39:23.277414 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-0e937d54-3d41-4164-a304-cfd5284aba47 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:39:23.277414 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:23.277414 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:23.277414 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:23.277414 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:23.277414 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:23.277414 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:23.277414 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:23.277414 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:23.277414 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:23.277414 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:23.277414 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:23.277414 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:39:23.277414 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:39:23.277414 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:39:23.277414 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:39:23.278295 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:39:23.278295 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:39:23.278295 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:39:23.278295 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:39:23.278295 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:23.278295 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:23.278295 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:23.278295 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:23.278295 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:23.278295 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:23.278295 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:23.278295 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:23.278295 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:23.278295 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:23.278295 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:23.278295 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:23.279392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:23.279392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:23.279392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:23.279392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:23.279392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:39:23.279392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:39:23.279392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:39:23.279392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:39:23.279392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:39:23.279392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:39:23.279392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:39:23.279392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:23.279392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 217/440] 10.222.0.27 () {56 vars in 1096 bytes} [Sun Sep 11 00:39:23 2022] GET /identity/v3/roles/admin => generated 83 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:39:23.279392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e937d54-3d41-4164-a304-cfd5284aba47 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:23.284269 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f90f244-fd9e-4fbc-b144-ab6cb568ac23 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:23.286016 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f90f244-fd9e-4fbc-b144-ab6cb568ac23 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:23.337651 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:23.344840 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:23.381222 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e937d54-3d41-4164-a304-cfd5284aba47 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:23.382204 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 224/441] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:39:23 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 327 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 11 00:39:23.383634 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f90f244-fd9e-4fbc-b144-ab6cb568ac23 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:23.416304 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b0bd024-28ef-45b6-a1ab-f6c0b54bf1fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:23.416940 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0bd024-28ef-45b6-a1ab-f6c0b54bf1fc None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:23.417221 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0bd024-28ef-45b6-a1ab-f6c0b54bf1fc None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:23.417489 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0bd024-28ef-45b6-a1ab-f6c0b54bf1fc None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:23.427608 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b0bd024-28ef-45b6-a1ab-f6c0b54bf1fc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:39:23.432579 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-0b0bd024-28ef-45b6-a1ab-f6c0b54bf1fc None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 11 00:39:23.432579 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:23.432579 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:23.432579 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:23.432579 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:23.432579 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:23.432579 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:23.432579 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:23.432579 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:23.432579 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:23.432579 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:23.432579 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:23.432579 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:39:23.432579 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:39:23.432579 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:39:23.432579 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:39:23.433694 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:39:23.433694 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:39:23.433694 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:39:23.433694 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:39:23.433694 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:23.433694 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:23.433694 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:23.433694 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:23.433694 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:23.433694 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:23.433694 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:23.433694 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:23.433694 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:23.433694 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:23.433694 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:23.433694 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:23.434544 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:23.434544 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:23.434544 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:23.434544 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:23.434544 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:39:23.434544 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:39:23.434544 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:39:23.434544 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:39:23.434544 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:39:23.434544 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:39:23.434544 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 11 00:39:23.434544 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:23.434544 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 218/442] 10.222.0.27 () {56 vars in 1114 bytes} [Sun Sep 11 00:39:23 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:23.434544 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b0bd024-28ef-45b6-a1ab-f6c0b54bf1fc None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:23.482422 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f90f244-fd9e-4fbc-b144-ab6cb568ac23 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:23.516062 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d5c2273-c001-4e36-b87b-a218dc875303 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:23.516657 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5c2273-c001-4e36-b87b-a218dc875303 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:23.516936 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5c2273-c001-4e36-b87b-a218dc875303 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:23.517206 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5c2273-c001-4e36-b87b-a218dc875303 None admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:23.517896 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d5c2273-c001-4e36-b87b-a218dc875303 None admin] RBAC: Adding query filter params (name=admin) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:23.518197 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d5c2273-c001-4e36-b87b-a218dc875303 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:23.520161 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d5c2273-c001-4e36-b87b-a218dc875303 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:23.529279 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 225/443] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:39:23 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:23.530082 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d5c2273-c001-4e36-b87b-a218dc875303 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:23.538926 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b0bd024-28ef-45b6-a1ab-f6c0b54bf1fc None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:23.556337 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:23.558046 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:23.574376 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2674bf70-58b0-44ab-84da-9cda7f117590 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:23.575031 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2674bf70-58b0-44ab-84da-9cda7f117590 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:23.575306 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2674bf70-58b0-44ab-84da-9cda7f117590 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:23.575576 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2674bf70-58b0-44ab-84da-9cda7f117590 None admin] PATH_INFO: `/v3/projects/a3af44366c3a450994b868f0b5428c43` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:23.576693 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2674bf70-58b0-44ab-84da-9cda7f117590 None admin] RBAC: Authorizing `identity:get_project(project_id=a3af44366c3a450994b868f0b5428c43)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:23.578143 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2674bf70-58b0-44ab-84da-9cda7f117590 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:23.579783 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 219/444] 10.222.0.27 () {56 vars in 1186 bytes} [Sun Sep 11 00:39:23 2022] GET /identity/v3/projects/a3af44366c3a450994b868f0b5428c43 => generated 302 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:23.580532 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2674bf70-58b0-44ab-84da-9cda7f117590 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:23.635049 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d5c2273-c001-4e36-b87b-a218dc875303 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:23.669128 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4806e354-c84d-44b8-8c8f-74db3bbf22aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:23.669729 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4806e354-c84d-44b8-8c8f-74db3bbf22aa None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:23.670011 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4806e354-c84d-44b8-8c8f-74db3bbf22aa None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:23.670282 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4806e354-c84d-44b8-8c8f-74db3bbf22aa None admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:23.670982 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4806e354-c84d-44b8-8c8f-74db3bbf22aa None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:23.671284 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4806e354-c84d-44b8-8c8f-74db3bbf22aa None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:23.672646 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4806e354-c84d-44b8-8c8f-74db3bbf22aa None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:23.679009 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2674bf70-58b0-44ab-84da-9cda7f117590 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:23.680336 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 226/445] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:39:23 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:23.681104 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4806e354-c84d-44b8-8c8f-74db3bbf22aa None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:23.721885 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c647c5d-31ea-4ffb-a39d-f9c068339ca6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:23.722462 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c647c5d-31ea-4ffb-a39d-f9c068339ca6 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:23.722745 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c647c5d-31ea-4ffb-a39d-f9c068339ca6 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:23.723049 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c647c5d-31ea-4ffb-a39d-f9c068339ca6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:23.775974 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:23.777293 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:23.790158 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4806e354-c84d-44b8-8c8f-74db3bbf22aa None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:23.790518 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c647c5d-31ea-4ffb-a39d-f9c068339ca6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:23.792023 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c647c5d-31ea-4ffb-a39d-f9c068339ca6 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:23.823737 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-150c8274-7f9b-40ac-b4ab-7ffb9f8e83fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:23.824363 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150c8274-7f9b-40ac-b4ab-7ffb9f8e83fd None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:23.824651 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150c8274-7f9b-40ac-b4ab-7ffb9f8e83fd None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:23.824921 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150c8274-7f9b-40ac-b4ab-7ffb9f8e83fd None admin] PATH_INFO: `/v3/groups/3a2cf5af1fa9407ca3402e9d17116e18` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:23.826042 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-150c8274-7f9b-40ac-b4ab-7ffb9f8e83fd None admin] RBAC: Authorizing `identity:get_group(group_id=3a2cf5af1fa9407ca3402e9d17116e18)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:23.827590 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-150c8274-7f9b-40ac-b4ab-7ffb9f8e83fd None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:23.829145 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 227/446] 10.222.0.27 () {56 vars in 1180 bytes} [Sun Sep 11 00:39:23 2022] GET /identity/v3/groups/3a2cf5af1fa9407ca3402e9d17116e18 => generated 226 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:23.829901 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-150c8274-7f9b-40ac-b4ab-7ffb9f8e83fd None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:23.894871 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 220/447] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:39:23 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 11 00:39:23.895969 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c647c5d-31ea-4ffb-a39d-f9c068339ca6 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:23.939074 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-150c8274-7f9b-40ac-b4ab-7ffb9f8e83fd None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:23.972648 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49136a25-269d-4b97-97f6-8667a3f88bce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:23.973249 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49136a25-269d-4b97-97f6-8667a3f88bce None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:23.973528 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49136a25-269d-4b97-97f6-8667a3f88bce None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:23.973792 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49136a25-269d-4b97-97f6-8667a3f88bce None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:23.978845 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:23.994071 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c647c5d-31ea-4ffb-a39d-f9c068339ca6 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:23.998763 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:24.027989 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10207e8-52ce-40da-9c3b-a77b1ed6ec5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:24.028512 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10207e8-52ce-40da-9c3b-a77b1ed6ec5d None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:24.028792 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10207e8-52ce-40da-9c3b-a77b1ed6ec5d None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:24.029064 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10207e8-52ce-40da-9c3b-a77b1ed6ec5d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:24.029782 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d10207e8-52ce-40da-9c3b-a77b1ed6ec5d None admin] RBAC: Adding query filter params (group.id=3a2cf5af1fa9407ca3402e9d17116e18, role.id=2e64fc597b6447df99a7992d411d1a34, scope.project.id=a3af44366c3a450994b868f0b5428c43) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:24.030087 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d10207e8-52ce-40da-9c3b-a77b1ed6ec5d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:24.031538 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d10207e8-52ce-40da-9c3b-a77b1ed6ec5d None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:24.037437 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 221/448] 10.222.0.27 () {56 vars in 1376 bytes} [Sun Sep 11 00:39:23 2022] GET /identity/v3/role_assignments?group.id=3a2cf5af1fa9407ca3402e9d17116e18&scope.project.id=a3af44366c3a450994b868f0b5428c43&role.id=2e64fc597b6447df99a7992d411d1a34 => generated 611 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:24.038196 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10207e8-52ce-40da-9c3b-a77b1ed6ec5d None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:24.042718 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49136a25-269d-4b97-97f6-8667a3f88bce None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:24.044233 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49136a25-269d-4b97-97f6-8667a3f88bce None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:24.161857 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 228/449] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:39:23 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 332 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 11 00:39:24.168796 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49136a25-269d-4b97-97f6-8667a3f88bce None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:24.175474 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10207e8-52ce-40da-9c3b-a77b1ed6ec5d None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:24.216291 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee258f3b-1ba2-4c2a-a227-7279ddf9de05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:24.216651 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee258f3b-1ba2-4c2a-a227-7279ddf9de05 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:24.216932 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee258f3b-1ba2-4c2a-a227-7279ddf9de05 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:24.217205 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee258f3b-1ba2-4c2a-a227-7279ddf9de05 None admin] PATH_INFO: `/v3/projects/12356f1769944cc8be5c533d062adf65` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:24.218342 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee258f3b-1ba2-4c2a-a227-7279ddf9de05 None admin] RBAC: Authorizing `identity:get_project(project_id=12356f1769944cc8be5c533d062adf65)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:24.219830 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee258f3b-1ba2-4c2a-a227-7279ddf9de05 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:24.221455 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 222/450] 10.222.0.27 () {56 vars in 1186 bytes} [Sun Sep 11 00:39:24 2022] GET /identity/v3/projects/12356f1769944cc8be5c533d062adf65 => generated 344 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:24.227596 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee258f3b-1ba2-4c2a-a227-7279ddf9de05 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:24.232604 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:24.255555 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:24.292502 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49136a25-269d-4b97-97f6-8667a3f88bce None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:24.333282 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7195d2c3-a219-4cd0-b28d-b5a82a4c309d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:24.334245 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7195d2c3-a219-4cd0-b28d-b5a82a4c309d None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:24.334520 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7195d2c3-a219-4cd0-b28d-b5a82a4c309d None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:24.334809 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7195d2c3-a219-4cd0-b28d-b5a82a4c309d None admin] PATH_INFO: `/v3/projects/a3af44366c3a450994b868f0b5428c43` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:24.336002 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7195d2c3-a219-4cd0-b28d-b5a82a4c309d None admin] RBAC: Authorizing `identity:get_project(project_id=a3af44366c3a450994b868f0b5428c43)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:24.338203 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7195d2c3-a219-4cd0-b28d-b5a82a4c309d None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:24.344611 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 229/451] 10.222.0.27 () {56 vars in 1186 bytes} [Sun Sep 11 00:39:24 2022] GET /identity/v3/projects/a3af44366c3a450994b868f0b5428c43 => generated 302 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:39:24.345851 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7195d2c3-a219-4cd0-b28d-b5a82a4c309d None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:24.359336 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee258f3b-1ba2-4c2a-a227-7279ddf9de05 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:24.403979 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a933e1c5-cb61-4752-b4a6-3f1ad6e2250a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:24.404639 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a933e1c5-cb61-4752-b4a6-3f1ad6e2250a None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:24.405070 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a933e1c5-cb61-4752-b4a6-3f1ad6e2250a None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:24.405276 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a933e1c5-cb61-4752-b4a6-3f1ad6e2250a None admin] PATH_INFO: `/v3/groups/dbc01bc745ca46db93dee6ac3ec2c72e` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:24.406437 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a933e1c5-cb61-4752-b4a6-3f1ad6e2250a None admin] RBAC: Authorizing `identity:get_group(group_id=dbc01bc745ca46db93dee6ac3ec2c72e)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:24.408057 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a933e1c5-cb61-4752-b4a6-3f1ad6e2250a None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:24.409661 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 223/452] 10.222.0.27 () {56 vars in 1180 bytes} [Sun Sep 11 00:39:24 2022] GET /identity/v3/groups/dbc01bc745ca46db93dee6ac3ec2c72e => generated 229 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:24.416449 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a933e1c5-cb61-4752-b4a6-3f1ad6e2250a None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:24.482070 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7195d2c3-a219-4cd0-b28d-b5a82a4c309d None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:24.507305 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:24.514252 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:24.519075 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57c99ffc-7589-4e43-94d3-175947961928 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:24.519940 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c99ffc-7589-4e43-94d3-175947961928 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:24.520218 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c99ffc-7589-4e43-94d3-175947961928 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:24.520488 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c99ffc-7589-4e43-94d3-175947961928 None admin] PATH_INFO: `/v3/groups/3a2cf5af1fa9407ca3402e9d17116e18` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:24.521628 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57c99ffc-7589-4e43-94d3-175947961928 None admin] RBAC: Authorizing `identity:get_group(group_id=3a2cf5af1fa9407ca3402e9d17116e18)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:24.523085 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57c99ffc-7589-4e43-94d3-175947961928 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:24.525726 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 230/453] 10.222.0.27 () {56 vars in 1180 bytes} [Sun Sep 11 00:39:24 2022] GET /identity/v3/groups/3a2cf5af1fa9407ca3402e9d17116e18 => generated 226 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:39:24.532188 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57c99ffc-7589-4e43-94d3-175947961928 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:24.532350 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a933e1c5-cb61-4752-b4a6-3f1ad6e2250a None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:24.577090 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7664bd06-9343-4895-ac9c-57d45d8d8c36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:24.579011 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7664bd06-9343-4895-ac9c-57d45d8d8c36 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:24.579303 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7664bd06-9343-4895-ac9c-57d45d8d8c36 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:24.579578 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7664bd06-9343-4895-ac9c-57d45d8d8c36 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:24.580317 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7664bd06-9343-4895-ac9c-57d45d8d8c36 None admin] RBAC: Adding query filter params (group.id=dbc01bc745ca46db93dee6ac3ec2c72e, role.id=7d9213c1187c4b5182bf8132cf23ea19, scope.project.id=12356f1769944cc8be5c533d062adf65) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:24.580625 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7664bd06-9343-4895-ac9c-57d45d8d8c36 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:24.582084 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7664bd06-9343-4895-ac9c-57d45d8d8c36 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:24.589750 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 224/454] 10.222.0.27 () {56 vars in 1376 bytes} [Sun Sep 11 00:39:24 2022] GET /identity/v3/role_assignments?group.id=dbc01bc745ca46db93dee6ac3ec2c72e&scope.project.id=12356f1769944cc8be5c533d062adf65&role.id=7d9213c1187c4b5182bf8132cf23ea19 => generated 611 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:39:24.642828 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57c99ffc-7589-4e43-94d3-175947961928 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:24.681704 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49980cd4-6f40-4526-ba94-047f54d9891e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:24.682326 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49980cd4-6f40-4526-ba94-047f54d9891e None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:24.682604 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49980cd4-6f40-4526-ba94-047f54d9891e None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:24.682904 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49980cd4-6f40-4526-ba94-047f54d9891e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:24.683646 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49980cd4-6f40-4526-ba94-047f54d9891e None admin] RBAC: Adding query filter params (group.id=3a2cf5af1fa9407ca3402e9d17116e18, role.id=518eab0f758241369edcdd997b98ad12, scope.project.id=a3af44366c3a450994b868f0b5428c43) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:24.683957 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49980cd4-6f40-4526-ba94-047f54d9891e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:24.685447 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49980cd4-6f40-4526-ba94-047f54d9891e None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:24.692252 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 231/455] 10.222.0.27 () {56 vars in 1376 bytes} [Sun Sep 11 00:39:24 2022] GET /identity/v3/role_assignments?group.id=3a2cf5af1fa9407ca3402e9d17116e18&scope.project.id=a3af44366c3a450994b868f0b5428c43&role.id=518eab0f758241369edcdd997b98ad12 => generated 611 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:39:27.870201 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e215865-5089-4855-a8b8-0e6373ba0ee0 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:27.872743 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e215865-5089-4855-a8b8-0e6373ba0ee0 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:27.872743 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e215865-5089-4855-a8b8-0e6373ba0ee0 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:27.873249 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 225/456] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:39:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:39:27.878151 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c58922a-9bda-4fdc-bfa5-7db3c4d9f9fb None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:27.878980 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c58922a-9bda-4fdc-bfa5-7db3c4d9f9fb None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:27.879468 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c58922a-9bda-4fdc-bfa5-7db3c4d9f9fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:27.907841 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-9c58922a-9bda-4fdc-bfa5-7db3c4d9f9fb None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:39:27.909351 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-9c58922a-9bda-4fdc-bfa5-7db3c4d9f9fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:39:27.979864 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:28.004431 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 232/457] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:39:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:39:28.145656 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee1f12ab-240e-4048-9bc2-8024ff536430 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:28.146241 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee1f12ab-240e-4048-9bc2-8024ff536430 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:28.146708 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee1f12ab-240e-4048-9bc2-8024ff536430 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:28.148600 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 226/458] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:39:28 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:39:28.155140 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa50684d-db2f-443f-8b73-63e1b0d3fc9e None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:28.155699 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa50684d-db2f-443f-8b73-63e1b0d3fc9e None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:28.156179 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa50684d-db2f-443f-8b73-63e1b0d3fc9e None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:28.157857 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 227/459] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:39:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 11 00:39:28.158576 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00060303-fa7f-4bad-a1dc-d81521bd35b6 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:28.159091 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00060303-fa7f-4bad-a1dc-d81521bd35b6 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:28.159552 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00060303-fa7f-4bad-a1dc-d81521bd35b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:28.162557 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbecb52d-ffc7-4bd0-9969-7f1f8a8c0de5 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:28.163075 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbecb52d-ffc7-4bd0-9969-7f1f8a8c0de5 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:28.163537 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbecb52d-ffc7-4bd0-9969-7f1f8a8c0de5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:28.192953 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-fbecb52d-ffc7-4bd0-9969-7f1f8a8c0de5 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:39:28.197961 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-fbecb52d-ffc7-4bd0-9969-7f1f8a8c0de5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:39:28.200946 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-00060303-fa7f-4bad-a1dc-d81521bd35b6 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:39:28.202294 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-00060303-fa7f-4bad-a1dc-d81521bd35b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:39:28.305897 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 228/460] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:39:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 11 00:39:28.333787 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 233/461] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:39:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:39:28.395954 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbecb52d-ffc7-4bd0-9969-7f1f8a8c0de5 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:28.458825 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-278db4be-cd99-477c-a292-eec8262256dc None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:28.458825 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-278db4be-cd99-477c-a292-eec8262256dc None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:28.458825 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-278db4be-cd99-477c-a292-eec8262256dc None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:28.460770 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 234/462] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:39:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:39:28.469999 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073164ff-c846-4150-b0b0-cd23a23c40c9 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:28.470570 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073164ff-c846-4150-b0b0-cd23a23c40c9 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:28.471086 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073164ff-c846-4150-b0b0-cd23a23c40c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:28.499800 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-073164ff-c846-4150-b0b0-cd23a23c40c9 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:39:28.501340 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-073164ff-c846-4150-b0b0-cd23a23c40c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:39:28.503418 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:28.537894 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbecb52d-ffc7-4bd0-9969-7f1f8a8c0de5 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:28.597573 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 235/463] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:39:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 11 00:39:28.609062 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17f4ab4c-397f-4199-969f-4c0f1e53ac06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:28.609704 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f4ab4c-397f-4199-969f-4c0f1e53ac06 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:28.609982 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f4ab4c-397f-4199-969f-4c0f1e53ac06 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:28.610254 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f4ab4c-397f-4199-969f-4c0f1e53ac06 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:28.626540 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17f4ab4c-397f-4199-969f-4c0f1e53ac06 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:28.628048 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17f4ab4c-397f-4199-969f-4c0f1e53ac06 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:28.631140 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-073164ff-c846-4150-b0b0-cd23a23c40c9 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:28.648493 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-17f4ab4c-397f-4199-969f-4c0f1e53ac06 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:28.648493 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:28.648493 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:28.648493 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:28.648493 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:28.648493 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:28.648493 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:28.648493 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:28.648493 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:28.648493 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:28.648493 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:28.648493 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:28.648493 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:39:28.648493 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:39:28.648493 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:39:28.648493 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:39:28.649625 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:28.649625 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:28.649625 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:28.649625 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:28.649625 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:28.649625 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:28.649625 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:28.649625 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:28.649625 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:28.649625 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:28.649625 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:28.649625 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:28.649625 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:28.649625 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:28.649625 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:28.649625 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:28.649625 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:39:28.650509 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:39:28.650509 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:28.650509 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:28.652121 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 229/464] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:39:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 257 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:28.653291 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17f4ab4c-397f-4199-969f-4c0f1e53ac06 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:28.707404 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:28.749079 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-073164ff-c846-4150-b0b0-cd23a23c40c9 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:28.773201 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17f4ab4c-397f-4199-969f-4c0f1e53ac06 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:28.787875 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8e2d4c-63c9-454a-b3f4-c883c2616bc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:28.788650 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8e2d4c-63c9-454a-b3f4-c883c2616bc0 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:28.789521 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8e2d4c-63c9-454a-b3f4-c883c2616bc0 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:28.789896 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8e2d4c-63c9-454a-b3f4-c883c2616bc0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:28.801846 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8e2d4c-63c9-454a-b3f4-c883c2616bc0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:28.803332 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8e2d4c-63c9-454a-b3f4-c883c2616bc0 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:28.807483 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:28.813495 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe2c25af-0fee-4cf3-9c49-2e3b3496a992 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:28.814234 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2c25af-0fee-4cf3-9c49-2e3b3496a992 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:28.814879 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2c25af-0fee-4cf3-9c49-2e3b3496a992 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:28.815647 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2c25af-0fee-4cf3-9c49-2e3b3496a992 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:28.822586 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-4f8e2d4c-63c9-454a-b3f4-c883c2616bc0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:28.822586 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:28.822586 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:28.822586 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:28.822586 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:28.822586 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:28.822586 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:28.822586 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:28.822586 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:28.822586 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:28.822586 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:28.822586 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:28.822586 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:39:28.822586 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:39:28.822586 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:39:28.822586 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:39:28.823506 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:28.823506 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:28.823506 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:28.823506 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:28.823506 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:28.823506 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:28.823506 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:28.823506 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:28.823506 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:28.823506 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:28.823506 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:28.823506 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:28.823506 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:28.823506 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:28.823506 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:28.823506 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:28.823506 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:39:28.824376 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:39:28.824376 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:28.824376 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:28.824376 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 236/465] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:39:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:39:28.829896 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8e2d4c-63c9-454a-b3f4-c883c2616bc0 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:28.835152 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe2c25af-0fee-4cf3-9c49-2e3b3496a992 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:28.837152 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe2c25af-0fee-4cf3-9c49-2e3b3496a992 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:28.853108 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-fe2c25af-0fee-4cf3-9c49-2e3b3496a992 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:28.853108 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:28.853108 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:28.853108 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:28.853108 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:28.853108 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:28.853108 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:28.853108 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:28.853108 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:28.853108 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:28.853108 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:28.853108 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:28.853108 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:39:28.853108 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:39:28.853108 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:39:28.853108 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:39:28.854059 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:28.854059 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:28.854059 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:28.854059 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:28.854059 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:28.854059 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:28.854059 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:28.854059 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:28.854059 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:28.854059 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:28.854059 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:28.854059 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:28.854059 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:28.854059 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:28.854059 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:28.854059 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:28.854059 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:39:28.854948 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:39:28.854948 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:28.854948 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:28.857057 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 230/466] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:39:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:28.858024 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe2c25af-0fee-4cf3-9c49-2e3b3496a992 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:28.937573 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:28.941780 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8e2d4c-63c9-454a-b3f4-c883c2616bc0 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:28.966219 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe2c25af-0fee-4cf3-9c49-2e3b3496a992 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:28.978569 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a467a2b7-2120-4d4b-b5be-cd370be16b87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:28.979269 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a467a2b7-2120-4d4b-b5be-cd370be16b87 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:28.979814 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a467a2b7-2120-4d4b-b5be-cd370be16b87 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:28.980087 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a467a2b7-2120-4d4b-b5be-cd370be16b87 None admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:28.981115 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a467a2b7-2120-4d4b-b5be-cd370be16b87 None admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:28.981429 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a467a2b7-2120-4d4b-b5be-cd370be16b87 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:28.982893 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a467a2b7-2120-4d4b-b5be-cd370be16b87 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:28.999889 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 237/467] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:39:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:29.000781 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a467a2b7-2120-4d4b-b5be-cd370be16b87 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:29.001414 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a704def8-c1db-4750-b074-e92215c1d337 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:29.002215 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a704def8-c1db-4750-b074-e92215c1d337 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:29.002672 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a704def8-c1db-4750-b074-e92215c1d337 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:29.003149 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a704def8-c1db-4750-b074-e92215c1d337 None admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:29.003988 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a704def8-c1db-4750-b074-e92215c1d337 None admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:29.004479 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a704def8-c1db-4750-b074-e92215c1d337 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:29.006028 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a704def8-c1db-4750-b074-e92215c1d337 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:29.020021 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 231/468] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:39:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:29.021149 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a704def8-c1db-4750-b074-e92215c1d337 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:29.066513 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:29.110538 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a467a2b7-2120-4d4b-b5be-cd370be16b87 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:29.124453 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a704def8-c1db-4750-b074-e92215c1d337 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:29.144028 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbeb54e5-f90d-4f95-a626-381e810e9c73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:29.144630 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbeb54e5-f90d-4f95-a626-381e810e9c73 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:29.145334 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbeb54e5-f90d-4f95-a626-381e810e9c73 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:29.145612 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbeb54e5-f90d-4f95-a626-381e810e9c73 None admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:29.146259 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbeb54e5-f90d-4f95-a626-381e810e9c73 None admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:29.146569 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbeb54e5-f90d-4f95-a626-381e810e9c73 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:29.147987 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbeb54e5-f90d-4f95-a626-381e810e9c73 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:29.159308 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b068d89e-81a1-410a-9fca-1e716d6b574c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:29.159447 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 238/469] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:39:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:29.159930 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b068d89e-81a1-410a-9fca-1e716d6b574c None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:29.160206 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b068d89e-81a1-410a-9fca-1e716d6b574c None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:29.160735 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b068d89e-81a1-410a-9fca-1e716d6b574c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:29.165586 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbeb54e5-f90d-4f95-a626-381e810e9c73 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:29.171551 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b068d89e-81a1-410a-9fca-1e716d6b574c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:29.173045 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b068d89e-81a1-410a-9fca-1e716d6b574c None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:29.183901 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:29.187663 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-b068d89e-81a1-410a-9fca-1e716d6b574c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:29.187663 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:29.187663 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:29.187663 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:29.187663 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:29.187663 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:29.187663 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:29.187663 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:29.187663 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:29.187663 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:29.187663 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:29.187663 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:29.187663 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:39:29.187663 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:39:29.187663 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:39:29.187663 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:39:29.188718 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:29.188718 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:29.188718 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:29.188718 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:29.188718 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:29.188718 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:29.188718 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:29.188718 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:29.188718 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:29.188718 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:29.188718 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:29.188718 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:29.188718 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:29.188718 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:29.188718 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:29.188718 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:29.188718 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:39:29.189769 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:39:29.189769 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:29.189769 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:29.189769 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 232/470] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:39:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:29.190003 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b068d89e-81a1-410a-9fca-1e716d6b574c None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:29.270301 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbeb54e5-f90d-4f95-a626-381e810e9c73 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:29.284637 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:29.291516 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b068d89e-81a1-410a-9fca-1e716d6b574c None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:29.304340 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-447e554a-c94c-4833-827a-34e32af6b875 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:29.305485 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447e554a-c94c-4833-827a-34e32af6b875 None admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:29.305814 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447e554a-c94c-4833-827a-34e32af6b875 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:29.306022 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447e554a-c94c-4833-827a-34e32af6b875 None admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:29.306665 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-447e554a-c94c-4833-827a-34e32af6b875 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:29.308128 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-447e554a-c94c-4833-827a-34e32af6b875 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:29.326396 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e4e77f2-4e45-4d63-9716-d0ecd12c0260 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:29.327028 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4e77f2-4e45-4d63-9716-d0ecd12c0260 None admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:29.327308 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4e77f2-4e45-4d63-9716-d0ecd12c0260 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:29.327582 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4e77f2-4e45-4d63-9716-d0ecd12c0260 None admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:29.328224 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e4e77f2-4e45-4d63-9716-d0ecd12c0260 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:29.337318 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 239/471] 10.222.0.27 () {60 vars in 1132 bytes} [Sun Sep 11 00:39:29 2022] POST /identity/v3/users => generated 246 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:39:29.337544 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e4e77f2-4e45-4d63-9716-d0ecd12c0260 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:29.337628 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc8241c-70f6-4fb1-85a4-9b2a2deb8776 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:29.337628 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc8241c-70f6-4fb1-85a4-9b2a2deb8776 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:29.337628 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc8241c-70f6-4fb1-85a4-9b2a2deb8776 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:29.337628 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 240/472] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:39:29 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:39:29.337628 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc8241c-70f6-4fb1-85a4-9b2a2deb8776 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:29.359013 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 233/473] 10.222.0.27 () {60 vars in 1132 bytes} [Sun Sep 11 00:39:29 2022] POST /identity/v3/users => generated 248 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:39:29.360000 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e4e77f2-4e45-4d63-9716-d0ecd12c0260 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:29.454351 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:29.498251 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc8241c-70f6-4fb1-85a4-9b2a2deb8776 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:29.504511 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e4e77f2-4e45-4d63-9716-d0ecd12c0260 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:29.545552 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0210451b-372a-42a7-941e-ac24b7401046 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:29.546173 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0210451b-372a-42a7-941e-ac24b7401046 None admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:29.546452 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0210451b-372a-42a7-941e-ac24b7401046 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:29.546728 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0210451b-372a-42a7-941e-ac24b7401046 None admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:29.547423 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0210451b-372a-42a7-941e-ac24b7401046 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:29.547768 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4c74b26-051a-4e24-a9c5-207e4acd9691 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:29.548455 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c74b26-051a-4e24-a9c5-207e4acd9691 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:29.548823 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c74b26-051a-4e24-a9c5-207e4acd9691 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:29.548939 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0210451b-372a-42a7-941e-ac24b7401046 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:29.549163 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c74b26-051a-4e24-a9c5-207e4acd9691 None admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:29.549887 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4c74b26-051a-4e24-a9c5-207e4acd9691 None admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:29.550259 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4c74b26-051a-4e24-a9c5-207e4acd9691 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:29.551680 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4c74b26-051a-4e24-a9c5-207e4acd9691 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:29.567049 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 234/474] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:39:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:29.568224 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d3a517-4eed-4a84-ab7e-b276d2d1ce32 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:29.568506 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d3a517-4eed-4a84-ab7e-b276d2d1ce32 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:29.568776 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d3a517-4eed-4a84-ab7e-b276d2d1ce32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:29.574276 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 241/475] 10.222.0.27 () {60 vars in 1132 bytes} [Sun Sep 11 00:39:29 2022] POST /identity/v3/users => generated 248 bytes in 242 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:39:29.575389 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0210451b-372a-42a7-941e-ac24b7401046 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:29.591389 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-44d3a517-4eed-4a84-ab7e-b276d2d1ce32 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:39:29.592694 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-44d3a517-4eed-4a84-ab7e-b276d2d1ce32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:39:29.610448 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:29.690720 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 235/476] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:39:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:39:29.727999 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0210451b-372a-42a7-941e-ac24b7401046 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:29.760129 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:29.767895 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d79bfefc-af03-4208-b3fc-ea8334c7ceee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:29.768870 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d79bfefc-af03-4208-b3fc-ea8334c7ceee None admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:29.769350 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d79bfefc-af03-4208-b3fc-ea8334c7ceee None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:29.769798 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d79bfefc-af03-4208-b3fc-ea8334c7ceee None admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:29.770462 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d79bfefc-af03-4208-b3fc-ea8334c7ceee None admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:29.771933 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d79bfefc-af03-4208-b3fc-ea8334c7ceee None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:29.805936 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 242/477] 10.222.0.27 () {60 vars in 1132 bytes} [Sun Sep 11 00:39:29 2022] POST /identity/v3/users => generated 249 bytes in 231 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:39:29.971782 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d3a517-4eed-4a84-ab7e-b276d2d1ce32 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:30.093246 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d3a517-4eed-4a84-ab7e-b276d2d1ce32 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:30.129927 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccecac6e-2142-4c58-a3ca-29b88474163d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:30.130913 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccecac6e-2142-4c58-a3ca-29b88474163d None admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:30.131383 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccecac6e-2142-4c58-a3ca-29b88474163d None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:30.131844 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccecac6e-2142-4c58-a3ca-29b88474163d None admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:30.132759 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccecac6e-2142-4c58-a3ca-29b88474163d None admin] RBAC: Authorizing `identity:create_role()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:30.134708 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccecac6e-2142-4c58-a3ca-29b88474163d None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:30.145082 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.sql.core [None req-ccecac6e-2142-4c58-a3ca-29b88474163d None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 11 00:39:30.145082 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Sep 11 00:39:30.145082 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [parameters: {'id': '9aca2ea4dc4e4dd8ac45554962600554', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Sep 11 00:39:30.145082 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=87923) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 11 00:39:30.162856 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-ccecac6e-2142-4c58-a3ca-29b88474163d None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Sep 11 00:39:30.162856 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:30.162856 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 11 00:39:30.162856 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 11 00:39:30.162856 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 11 00:39:30.162856 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 11 00:39:30.162856 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 11 00:39:30.162856 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application result = self._query(query) Sep 11 00:39:30.162856 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 11 00:39:30.162856 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application conn.query(q) Sep 11 00:39:30.162856 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 11 00:39:30.162856 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 11 00:39:30.162856 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 11 00:39:30.162856 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application result.read() Sep 11 00:39:30.162856 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 11 00:39:30.163789 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 11 00:39:30.163789 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 11 00:39:30.163789 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 11 00:39:30.163789 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 11 00:39:30.163789 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 11 00:39:30.163789 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 11 00:39:30.163789 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 11 00:39:30.163789 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 11 00:39:30.163789 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:30.163789 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 11 00:39:30.163789 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:30.163789 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:30.163789 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 11 00:39:30.163789 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 11 00:39:30.163789 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Sep 11 00:39:30.163789 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return ref.to_dict() Sep 11 00:39:30.163789 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 11 00:39:30.163789 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application next(self.gen) Sep 11 00:39:30.165714 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 11 00:39:30.165714 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application yield resource Sep 11 00:39:30.165714 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 11 00:39:30.165714 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application next(self.gen) Sep 11 00:39:30.165714 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 11 00:39:30.165714 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application self.session.rollback() Sep 11 00:39:30.165714 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 11 00:39:30.165714 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application self.force_reraise() Sep 11 00:39:30.165714 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 11 00:39:30.165714 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise self.value Sep 11 00:39:30.165714 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 11 00:39:30.165714 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 11 00:39:30.165714 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 11 00:39:30.165714 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application session.commit() Sep 11 00:39:30.165714 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 11 00:39:30.165714 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 11 00:39:30.165714 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 11 00:39:30.166587 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application self._prepare_impl() Sep 11 00:39:30.166587 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 11 00:39:30.166587 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application self.session.flush() Sep 11 00:39:30.166587 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 11 00:39:30.166587 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application self._flush(objects) Sep 11 00:39:30.166587 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 11 00:39:30.166587 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 11 00:39:30.166587 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 11 00:39:30.166587 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application compat.raise_( Sep 11 00:39:30.166587 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 11 00:39:30.166587 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception Sep 11 00:39:30.166587 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 11 00:39:30.166587 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application flush_context.execute() Sep 11 00:39:30.166587 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 11 00:39:30.166587 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rec.execute(self) Sep 11 00:39:30.166587 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 11 00:39:30.166587 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 11 00:39:30.167486 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 11 00:39:30.167486 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 11 00:39:30.167486 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Sep 11 00:39:30.167486 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application c = connection._execute_20( Sep 11 00:39:30.167486 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 11 00:39:30.167486 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 11 00:39:30.167486 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 11 00:39:30.167486 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 11 00:39:30.167486 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 11 00:39:30.167486 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 11 00:39:30.167486 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 11 00:39:30.167486 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 11 00:39:30.167486 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 11 00:39:30.167486 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 11 00:39:30.167486 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 11 00:39:30.167486 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception Sep 11 00:39:30.168305 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 11 00:39:30.168305 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 11 00:39:30.168305 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 11 00:39:30.168305 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 11 00:39:30.168305 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 11 00:39:30.168305 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application result = self._query(query) Sep 11 00:39:30.168305 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 11 00:39:30.168305 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application conn.query(q) Sep 11 00:39:30.168305 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 11 00:39:30.168305 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 11 00:39:30.168305 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 11 00:39:30.168305 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application result.read() Sep 11 00:39:30.168305 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 11 00:39:30.168305 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 11 00:39:30.168305 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 11 00:39:30.168305 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 11 00:39:30.169167 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 11 00:39:30.169167 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 11 00:39:30.169167 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 11 00:39:30.169167 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 11 00:39:30.169167 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 11 00:39:30.169167 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Sep 11 00:39:30.169167 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [parameters: {'id': '9aca2ea4dc4e4dd8ac45554962600554', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Sep 11 00:39:30.169167 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 11 00:39:30.169167 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:30.169167 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 11 00:39:30.169167 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:30.169167 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:30.169167 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:30.169167 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:30.169167 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:30.169167 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:30.169967 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:30.169967 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:30.169967 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:30.169967 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:30.169967 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:30.169967 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:30.169967 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Sep 11 00:39:30.169967 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Sep 11 00:39:30.169967 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:30.169967 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:30.169967 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Sep 11 00:39:30.169967 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Sep 11 00:39:30.169967 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 11 00:39:30.169967 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 11 00:39:30.169967 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Sep 11 00:39:30.169967 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:30.173887 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 236/478] 10.222.0.27 () {60 vars in 1131 bytes} [Sun Sep 11 00:39:29 2022] POST /identity/v3/roles => generated 144 bytes in 203 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:30.179473 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d79bfefc-af03-4208-b3fc-ea8334c7ceee None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:30.331404 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d79bfefc-af03-4208-b3fc-ea8334c7ceee None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:30.369535 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57b62374-b17e-454d-a24d-335924c17c94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:30.370523 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b62374-b17e-454d-a24d-335924c17c94 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:30.371030 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b62374-b17e-454d-a24d-335924c17c94 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:30.371493 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b62374-b17e-454d-a24d-335924c17c94 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:30.382618 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57b62374-b17e-454d-a24d-335924c17c94 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:39:30.389342 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-57b62374-b17e-454d-a24d-335924c17c94 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 11 00:39:30.389342 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:30.389342 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:30.389342 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:30.389342 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:30.389342 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:30.389342 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:30.389342 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:30.389342 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:30.389342 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:30.389342 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:30.389342 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:30.389342 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:39:30.389342 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:39:30.389342 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:39:30.389342 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:39:30.390229 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:39:30.390229 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:39:30.390229 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:39:30.390229 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:39:30.390229 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:30.390229 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:30.390229 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:30.390229 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:30.390229 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:30.390229 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:30.390229 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:30.390229 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:30.390229 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:30.390229 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:30.390229 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:30.390229 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:30.391107 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:30.391107 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:30.391107 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:30.391107 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:30.391107 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:39:30.391107 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:39:30.391107 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:39:30.391107 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:39:30.391107 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:39:30.391107 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:39:30.391107 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 11 00:39:30.391107 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:30.393689 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 243/479] 10.222.0.27 () {56 vars in 1114 bytes} [Sun Sep 11 00:39:30 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:30.399737 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccecac6e-2142-4c58-a3ca-29b88474163d None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:30.412198 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:30.544965 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccecac6e-2142-4c58-a3ca-29b88474163d None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:30.593300 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87844c0d-1e38-4a02-8249-71ad06121f92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:30.594261 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87844c0d-1e38-4a02-8249-71ad06121f92 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:30.594733 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87844c0d-1e38-4a02-8249-71ad06121f92 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:30.595216 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87844c0d-1e38-4a02-8249-71ad06121f92 None admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:30.596066 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87844c0d-1e38-4a02-8249-71ad06121f92 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:30.596584 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87844c0d-1e38-4a02-8249-71ad06121f92 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:30.598135 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87844c0d-1e38-4a02-8249-71ad06121f92 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:30.611184 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 237/480] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:39:30 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:31.858629 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc17f90-2640-4e45-87cb-ae69e7b9aace None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:31.859202 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc17f90-2640-4e45-87cb-ae69e7b9aace None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:31.859202 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc17f90-2640-4e45-87cb-ae69e7b9aace None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:31.860753 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 244/481] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:39:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:39:31.866404 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb8daa2-e499-496d-9beb-fa6d4f638fd6 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:31.866848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb8daa2-e499-496d-9beb-fa6d4f638fd6 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:31.867234 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb8daa2-e499-496d-9beb-fa6d4f638fd6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:31.892303 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-6cb8daa2-e499-496d-9beb-fa6d4f638fd6 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:39:31.893818 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-6cb8daa2-e499-496d-9beb-fa6d4f638fd6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:39:31.985298 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:31.997322 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 238/482] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:39:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:39:32.227076 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1fef0ea-c0b2-43da-8669-82690f4f311e None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:32.227553 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1fef0ea-c0b2-43da-8669-82690f4f311e None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:32.227972 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1fef0ea-c0b2-43da-8669-82690f4f311e None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:32.230255 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 245/483] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:39:32 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 11 00:39:32.238095 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32430c01-3455-4796-859e-34d53358668b None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:32.238599 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32430c01-3455-4796-859e-34d53358668b None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:32.239137 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32430c01-3455-4796-859e-34d53358668b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:32.273887 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-32430c01-3455-4796-859e-34d53358668b None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:39:32.275120 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-32430c01-3455-4796-859e-34d53358668b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:39:32.338880 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff9ad30-f561-4f6b-8942-656370a31178 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:32.339211 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff9ad30-f561-4f6b-8942-656370a31178 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:32.339425 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff9ad30-f561-4f6b-8942-656370a31178 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:32.341002 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 246/484] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:39:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 11 00:39:32.346161 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92733671-0d45-4de6-bcff-13326c494054 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:32.346445 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92733671-0d45-4de6-bcff-13326c494054 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:32.346719 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92733671-0d45-4de6-bcff-13326c494054 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:32.370061 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-92733671-0d45-4de6-bcff-13326c494054 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:39:32.371292 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-92733671-0d45-4de6-bcff-13326c494054 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:39:32.375977 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 239/485] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:39:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:39:32.376899 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32430c01-3455-4796-859e-34d53358668b None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:32.435160 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:32.441964 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 247/486] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:39:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:39:32.501485 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32430c01-3455-4796-859e-34d53358668b None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:32.529259 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:32.537958 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148ed8c1-327a-4b16-9ad6-7a4de4b687f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:32.538590 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148ed8c1-327a-4b16-9ad6-7a4de4b687f0 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:32.539433 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148ed8c1-327a-4b16-9ad6-7a4de4b687f0 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:32.539705 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148ed8c1-327a-4b16-9ad6-7a4de4b687f0 None admin] PATH_INFO: `/v3/roles/service` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:32.550220 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-148ed8c1-327a-4b16-9ad6-7a4de4b687f0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:39:32.555244 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-148ed8c1-327a-4b16-9ad6-7a4de4b687f0 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 11 00:39:32.555244 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:32.555244 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:32.555244 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:32.555244 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:32.555244 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:32.555244 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:32.555244 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:32.555244 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:32.555244 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:32.555244 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:32.555244 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:32.555244 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:39:32.555244 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:39:32.555244 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:39:32.555244 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:39:32.556125 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:39:32.556125 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:39:32.556125 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:39:32.556125 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:39:32.556125 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:32.556125 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:32.556125 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:32.556125 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:32.556125 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:32.556125 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:32.556125 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:32.556125 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:32.556125 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:32.556125 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:32.556125 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:32.556125 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:32.556926 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:32.556926 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:32.556926 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:32.556926 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:32.556926 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:39:32.556926 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:39:32.556926 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:39:32.556926 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:39:32.556926 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:39:32.556926 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:39:32.556926 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 11 00:39:32.556926 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:32.556926 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 240/487] 10.222.0.27 () {56 vars in 1102 bytes} [Sun Sep 11 00:39:32 2022] GET /identity/v3/roles/service => generated 85 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:32.562262 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92733671-0d45-4de6-bcff-13326c494054 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:32.681158 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148ed8c1-327a-4b16-9ad6-7a4de4b687f0 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:32.724486 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92733671-0d45-4de6-bcff-13326c494054 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:32.809660 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e199102b-72ec-472e-bd98-6ec91261286f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:32.811223 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e199102b-72ec-472e-bd98-6ec91261286f None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:32.811978 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e199102b-72ec-472e-bd98-6ec91261286f None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:32.812450 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e199102b-72ec-472e-bd98-6ec91261286f None admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:32.813607 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e199102b-72ec-472e-bd98-6ec91261286f None admin] RBAC: Adding query filter params (name=service) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:32.815081 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e199102b-72ec-472e-bd98-6ec91261286f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:32.816724 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e199102b-72ec-472e-bd98-6ec91261286f None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:32.828260 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 248/488] 10.222.0.27 () {56 vars in 1103 bytes} [Sun Sep 11 00:39:32 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 266 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:32.829339 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e199102b-72ec-472e-bd98-6ec91261286f None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:32.852834 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148ed8c1-327a-4b16-9ad6-7a4de4b687f0 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:32.881055 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:32.895696 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e928b1-13ff-4774-8df1-3d3addbccc43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:32.896736 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e928b1-13ff-4774-8df1-3d3addbccc43 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:32.896736 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e928b1-13ff-4774-8df1-3d3addbccc43 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:32.897005 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e928b1-13ff-4774-8df1-3d3addbccc43 None admin] PATH_INFO: `/v3/roles/service` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:32.909403 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14e928b1-13ff-4774-8df1-3d3addbccc43 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:39:32.914847 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-14e928b1-13ff-4774-8df1-3d3addbccc43 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 11 00:39:32.914847 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:32.914847 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:32.914847 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:32.914847 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:32.914847 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:32.914847 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:32.914847 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:32.914847 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:32.914847 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:32.914847 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:32.914847 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:32.914847 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:39:32.914847 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:39:32.914847 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:39:32.914847 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:39:32.915958 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:39:32.915958 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:39:32.915958 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:39:32.915958 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:39:32.915958 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:32.915958 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:32.915958 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:32.915958 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:32.915958 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:32.915958 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:32.915958 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:32.915958 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:32.915958 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:32.915958 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:32.915958 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:32.915958 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:32.922111 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:32.922111 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:32.922111 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:32.922111 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:32.922111 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:39:32.922111 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:39:32.922111 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:39:32.922111 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:39:32.922111 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:39:32.922111 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:39:32.922111 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 11 00:39:32.922111 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:32.922111 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 241/489] 10.222.0.27 () {56 vars in 1102 bytes} [Sun Sep 11 00:39:32 2022] GET /identity/v3/roles/service => generated 85 bytes in 235 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:39:32.922111 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e928b1-13ff-4774-8df1-3d3addbccc43 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:32.956846 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e199102b-72ec-472e-bd98-6ec91261286f None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:32.972020 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:32.992672 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec17d33-b478-4ccd-a988-c89258bb977e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:32.993375 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec17d33-b478-4ccd-a988-c89258bb977e None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:32.993654 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec17d33-b478-4ccd-a988-c89258bb977e None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:32.993924 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec17d33-b478-4ccd-a988-c89258bb977e None admin] PATH_INFO: `/v3/roles/service` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:33.005707 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ec17d33-b478-4ccd-a988-c89258bb977e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:39:33.011740 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-5ec17d33-b478-4ccd-a988-c89258bb977e None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 11 00:39:33.011740 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:33.011740 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:33.011740 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:33.011740 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:33.011740 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:33.011740 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:33.011740 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:33.011740 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:33.011740 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:33.011740 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:33.011740 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:33.011740 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:39:33.011740 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:39:33.011740 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:39:33.011740 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:39:33.012637 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:39:33.012637 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:39:33.012637 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:39:33.012637 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:39:33.012637 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:33.012637 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:33.012637 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:33.012637 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:33.012637 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:33.012637 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:33.012637 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:33.012637 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:33.012637 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:33.012637 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:33.012637 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:33.012637 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:33.013477 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:33.013477 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:33.013477 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:33.013477 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:33.013477 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:39:33.013477 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:39:33.013477 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:39:33.013477 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:39:33.013477 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:39:33.013477 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:39:33.013477 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 11 00:39:33.013477 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:33.013477 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 249/490] 10.222.0.27 () {56 vars in 1102 bytes} [Sun Sep 11 00:39:32 2022] GET /identity/v3/roles/service => generated 85 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:33.014169 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09bbc9dc-9f88-4768-adf6-8f9b900358e6 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:33.014893 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09bbc9dc-9f88-4768-adf6-8f9b900358e6 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:33.014893 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09bbc9dc-9f88-4768-adf6-8f9b900358e6 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:33.019080 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 250/491] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:39:33 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:39:33.019080 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09bbc9dc-9f88-4768-adf6-8f9b900358e6 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:33.031234 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e928b1-13ff-4774-8df1-3d3addbccc43 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:33.067503 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd231e1-85ef-4a40-818b-2c52e7a3a479 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:33.068164 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd231e1-85ef-4a40-818b-2c52e7a3a479 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:33.068439 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd231e1-85ef-4a40-818b-2c52e7a3a479 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:33.068820 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd231e1-85ef-4a40-818b-2c52e7a3a479 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:33.081617 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bd231e1-85ef-4a40-818b-2c52e7a3a479 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:33.083256 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bd231e1-85ef-4a40-818b-2c52e7a3a479 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:33.100842 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-7bd231e1-85ef-4a40-818b-2c52e7a3a479 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:33.100842 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:33.100842 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:33.100842 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:33.100842 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:33.100842 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:33.100842 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:33.100842 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:33.100842 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:33.100842 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:33.100842 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:33.100842 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:33.100842 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:39:33.100842 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:39:33.100842 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:39:33.100842 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:39:33.106743 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:33.106743 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:33.106743 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:33.106743 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:33.106743 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:33.106743 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:33.106743 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:33.106743 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:33.106743 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:33.106743 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:33.106743 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:33.106743 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:33.106743 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:33.106743 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:33.106743 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:33.106743 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:33.106743 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:39:33.108374 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:39:33.108374 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:33.108374 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:33.108374 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 242/492] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:39:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:33.108374 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd231e1-85ef-4a40-818b-2c52e7a3a479 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:33.131190 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:33.136850 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09bbc9dc-9f88-4768-adf6-8f9b900358e6 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:33.176267 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73c3776e-0083-41bf-986c-497ab6c849d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:33.176868 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c3776e-0083-41bf-986c-497ab6c849d5 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:33.177141 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c3776e-0083-41bf-986c-497ab6c849d5 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:33.177404 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c3776e-0083-41bf-986c-497ab6c849d5 None admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:33.178051 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73c3776e-0083-41bf-986c-497ab6c849d5 None admin] RBAC: Adding query filter params (name=service) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:33.178349 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73c3776e-0083-41bf-986c-497ab6c849d5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:33.179971 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73c3776e-0083-41bf-986c-497ab6c849d5 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:33.193208 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 251/493] 10.222.0.27 () {56 vars in 1103 bytes} [Sun Sep 11 00:39:33 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:33.193208 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00ea652-53fa-407f-93e3-ba69feaebebf None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:33.193208 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00ea652-53fa-407f-93e3-ba69feaebebf None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:33.193208 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00ea652-53fa-407f-93e3-ba69feaebebf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:33.209641 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-d00ea652-53fa-407f-93e3-ba69feaebebf None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:39:33.211054 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-d00ea652-53fa-407f-93e3-ba69feaebebf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:39:33.216599 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:33.220136 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd231e1-85ef-4a40-818b-2c52e7a3a479 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:33.258585 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67fe117d-15cd-4c02-981d-b9e83f3b2741 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:33.259223 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67fe117d-15cd-4c02-981d-b9e83f3b2741 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:33.259488 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67fe117d-15cd-4c02-981d-b9e83f3b2741 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:33.259761 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67fe117d-15cd-4c02-981d-b9e83f3b2741 None admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:33.260426 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67fe117d-15cd-4c02-981d-b9e83f3b2741 None admin] RBAC: Adding query filter params (name=service) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:33.260726 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67fe117d-15cd-4c02-981d-b9e83f3b2741 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:33.262201 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67fe117d-15cd-4c02-981d-b9e83f3b2741 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:33.274837 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 243/494] 10.222.0.27 () {56 vars in 1103 bytes} [Sun Sep 11 00:39:33 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:33.274837 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67fe117d-15cd-4c02-981d-b9e83f3b2741 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:33.294505 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 252/495] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:39:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:39:33.295663 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d00ea652-53fa-407f-93e3-ba69feaebebf None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:33.395283 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67fe117d-15cd-4c02-981d-b9e83f3b2741 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:33.407996 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:33.420236 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d00ea652-53fa-407f-93e3-ba69feaebebf None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:33.442197 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-442cb807-b2db-4f42-a004-141db9e54390 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:33.443225 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442cb807-b2db-4f42-a004-141db9e54390 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:33.444136 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442cb807-b2db-4f42-a004-141db9e54390 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:33.444613 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442cb807-b2db-4f42-a004-141db9e54390 None admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:33.446595 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-442cb807-b2db-4f42-a004-141db9e54390 None admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:33.447170 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-442cb807-b2db-4f42-a004-141db9e54390 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:33.449456 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-442cb807-b2db-4f42-a004-141db9e54390 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:33.461834 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9166c843-971a-474e-95a2-97a666d27c02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:33.462941 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9166c843-971a-474e-95a2-97a666d27c02 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:33.463239 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9166c843-971a-474e-95a2-97a666d27c02 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:33.463512 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9166c843-971a-474e-95a2-97a666d27c02 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:33.467278 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 244/496] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:39:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:33.468371 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-442cb807-b2db-4f42-a004-141db9e54390 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:33.479085 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9166c843-971a-474e-95a2-97a666d27c02 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:33.480453 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9166c843-971a-474e-95a2-97a666d27c02 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:33.496831 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-9166c843-971a-474e-95a2-97a666d27c02 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:33.496831 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:33.496831 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:33.496831 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:33.496831 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:33.496831 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:33.496831 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:33.496831 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:33.496831 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:33.496831 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:33.496831 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:33.496831 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:33.496831 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:39:33.496831 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:39:33.496831 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:39:33.496831 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:39:33.497797 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:33.497797 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:33.497797 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:33.497797 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:33.497797 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:33.497797 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:33.497797 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:33.497797 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:33.497797 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:33.497797 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:33.497797 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:33.497797 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:33.497797 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:33.497797 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:33.497797 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:33.497797 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:33.497797 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:39:33.498652 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:39:33.498652 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:33.498652 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:33.500046 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:33.502820 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 253/497] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:39:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:33.504104 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9166c843-971a-474e-95a2-97a666d27c02 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:33.596763 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-442cb807-b2db-4f42-a004-141db9e54390 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:33.621772 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9166c843-971a-474e-95a2-97a666d27c02 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:33.634643 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-740e2ac4-860f-4979-86f9-258cb90023e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:33.635301 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740e2ac4-860f-4979-86f9-258cb90023e9 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:33.635577 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740e2ac4-860f-4979-86f9-258cb90023e9 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:33.635846 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740e2ac4-860f-4979-86f9-258cb90023e9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:33.646025 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-740e2ac4-860f-4979-86f9-258cb90023e9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:33.647457 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-740e2ac4-860f-4979-86f9-258cb90023e9 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:33.656708 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff2d1b27-a289-4b79-8643-56608560161c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:33.657337 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2d1b27-a289-4b79-8643-56608560161c None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:33.657611 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2d1b27-a289-4b79-8643-56608560161c None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:33.657879 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2d1b27-a289-4b79-8643-56608560161c None admin] PATH_INFO: `/v3/users/nova` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:33.663633 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-740e2ac4-860f-4979-86f9-258cb90023e9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:33.663633 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:33.663633 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:33.663633 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:33.663633 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:33.663633 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:33.663633 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:33.663633 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:33.663633 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:33.663633 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:33.663633 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:33.663633 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:33.663633 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:39:33.663633 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:39:33.663633 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:39:33.663633 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:39:33.665237 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:33.665237 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:33.665237 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:33.665237 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:33.665237 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:33.665237 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:33.665237 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:33.665237 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:33.665237 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:33.665237 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:33.665237 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:33.665237 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:33.665237 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:33.665237 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:33.665237 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:33.665237 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:33.665237 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:39:33.666105 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:39:33.666105 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:33.666105 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:33.667124 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 245/498] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:39:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:33.668154 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-740e2ac4-860f-4979-86f9-258cb90023e9 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:33.671271 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff2d1b27-a289-4b79-8643-56608560161c None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:33.672718 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff2d1b27-a289-4b79-8643-56608560161c None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:33.686943 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-ff2d1b27-a289-4b79-8643-56608560161c None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 11 00:39:33.686943 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:33.686943 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:33.686943 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:33.686943 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:33.686943 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:33.686943 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:33.686943 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:33.686943 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:33.686943 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:33.686943 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:33.686943 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:33.686943 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 11 00:39:33.686943 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 11 00:39:33.686943 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 11 00:39:33.686943 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 11 00:39:33.687892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:33.687892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:33.687892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 11 00:39:33.687892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:39:33.687892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 11 00:39:33.687892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:39:33.687892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:33.687892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:33.687892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:33.687892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:33.687892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:33.687892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:33.687892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:33.687892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:33.687892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:33.687892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:33.687892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:33.687892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:33.688797 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:33.688797 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:33.688797 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 11 00:39:33.688797 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 11 00:39:33.688797 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 11 00:39:33.688797 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 11 00:39:33.688797 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 11 00:39:33.688797 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 11 00:39:33.688797 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 11 00:39:33.688797 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:33.688797 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 254/499] 10.222.0.27 () {56 vars in 1093 bytes} [Sun Sep 11 00:39:33 2022] GET /identity/v3/users/nova => generated 82 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:33.689382 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff2d1b27-a289-4b79-8643-56608560161c None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:33.691822 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:33.754589 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:33.783392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-740e2ac4-860f-4979-86f9-258cb90023e9 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:33.808593 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff2d1b27-a289-4b79-8643-56608560161c None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:33.818946 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc66cee-93bb-4d71-8897-e8553bc901d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:33.819855 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc66cee-93bb-4d71-8897-e8553bc901d6 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:33.820512 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc66cee-93bb-4d71-8897-e8553bc901d6 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:33.820786 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc66cee-93bb-4d71-8897-e8553bc901d6 None admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:33.821428 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dc66cee-93bb-4d71-8897-e8553bc901d6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:33.821742 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dc66cee-93bb-4d71-8897-e8553bc901d6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:33.823115 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dc66cee-93bb-4d71-8897-e8553bc901d6 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:33.834498 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 246/500] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:39:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:33.835331 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc66cee-93bb-4d71-8897-e8553bc901d6 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:33.850221 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca7fda5-61ec-4d2a-ad86-89f8c112979d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:33.851173 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca7fda5-61ec-4d2a-ad86-89f8c112979d None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:33.851947 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca7fda5-61ec-4d2a-ad86-89f8c112979d None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:33.852445 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca7fda5-61ec-4d2a-ad86-89f8c112979d None admin] PATH_INFO: `/v3/roles/service` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:33.863574 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ca7fda5-61ec-4d2a-ad86-89f8c112979d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:39:33.870162 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-9ca7fda5-61ec-4d2a-ad86-89f8c112979d None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 11 00:39:33.870162 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:33.870162 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:33.870162 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:33.870162 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:33.870162 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:33.870162 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:33.870162 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:33.870162 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:33.870162 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:33.870162 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:33.870162 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:33.870162 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:39:33.870162 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:39:33.870162 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:39:33.870162 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:39:33.871122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:39:33.871122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:39:33.871122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:39:33.871122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:39:33.871122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:33.871122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:33.871122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:33.871122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:33.871122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:33.871122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:33.871122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:33.871122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:33.871122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:33.871122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:33.871122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:33.871122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:33.872100 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:33.872100 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:33.872100 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:33.872100 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:33.872100 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:39:33.872100 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:39:33.872100 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:39:33.872100 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:39:33.872100 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:39:33.872100 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:39:33.872100 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 11 00:39:33.872100 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:33.874062 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 255/501] 10.222.0.27 () {56 vars in 1102 bytes} [Sun Sep 11 00:39:33 2022] GET /identity/v3/roles/service => generated 85 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:33.875002 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca7fda5-61ec-4d2a-ad86-89f8c112979d None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:33.938193 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:33.938810 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc66cee-93bb-4d71-8897-e8553bc901d6 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:33.973000 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71200dc1-f8ae-4511-aeb7-00bbe2157ebc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:33.973594 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71200dc1-f8ae-4511-aeb7-00bbe2157ebc None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:33.973870 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71200dc1-f8ae-4511-aeb7-00bbe2157ebc None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:33.974141 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71200dc1-f8ae-4511-aeb7-00bbe2157ebc None admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:33.974780 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71200dc1-f8ae-4511-aeb7-00bbe2157ebc None admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:33.975336 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71200dc1-f8ae-4511-aeb7-00bbe2157ebc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:33.977193 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71200dc1-f8ae-4511-aeb7-00bbe2157ebc None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:33.978590 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca7fda5-61ec-4d2a-ad86-89f8c112979d None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:33.978705 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:33.989247 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 247/502] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:39:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:33.990395 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71200dc1-f8ae-4511-aeb7-00bbe2157ebc None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:34.013266 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e08d14-be74-400a-b459-02a3271d5462 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:34.013868 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e08d14-be74-400a-b459-02a3271d5462 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:34.014145 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e08d14-be74-400a-b459-02a3271d5462 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:34.014414 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e08d14-be74-400a-b459-02a3271d5462 None admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:34.015169 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e08d14-be74-400a-b459-02a3271d5462 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:34.015473 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e08d14-be74-400a-b459-02a3271d5462 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:34.016905 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e08d14-be74-400a-b459-02a3271d5462 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:34.053276 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 256/503] 10.222.0.27 () {56 vars in 1133 bytes} [Sun Sep 11 00:39:33 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:34.055164 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b4f1a4-e08e-444f-971d-986043c96bfb None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:34.055452 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b4f1a4-e08e-444f-971d-986043c96bfb None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:34.055827 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b4f1a4-e08e-444f-971d-986043c96bfb None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:34.061365 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 257/504] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:39:34 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:39:34.062077 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b4f1a4-e08e-444f-971d-986043c96bfb None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:34.089724 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71200dc1-f8ae-4511-aeb7-00bbe2157ebc None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:34.125415 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5735e5f6-afc9-4f82-b63a-305d10394572 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:34.126037 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5735e5f6-afc9-4f82-b63a-305d10394572 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:34.126309 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5735e5f6-afc9-4f82-b63a-305d10394572 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:34.126575 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5735e5f6-afc9-4f82-b63a-305d10394572 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:34.133926 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5735e5f6-afc9-4f82-b63a-305d10394572 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:34.135384 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5735e5f6-afc9-4f82-b63a-305d10394572 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:34.147468 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-5735e5f6-afc9-4f82-b63a-305d10394572 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 11 00:39:34.147468 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:34.147468 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:34.147468 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:34.147468 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:34.147468 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:34.147468 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:34.147468 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:34.147468 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:34.147468 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:34.147468 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:34.147468 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:34.147468 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 11 00:39:34.147468 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 11 00:39:34.147468 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 11 00:39:34.147468 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 11 00:39:34.148654 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:34.148654 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:34.148654 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 11 00:39:34.148654 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:39:34.148654 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 11 00:39:34.148654 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:39:34.148654 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:34.148654 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:34.148654 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:34.148654 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:34.148654 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:34.148654 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:34.148654 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:34.148654 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:34.148654 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:34.148654 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:34.148654 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:34.148654 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:34.149557 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:34.149557 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:34.149557 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 11 00:39:34.149557 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 11 00:39:34.149557 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 11 00:39:34.149557 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 11 00:39:34.149557 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 11 00:39:34.149557 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 11 00:39:34.149557 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 11 00:39:34.149557 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:34.149557 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 248/505] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:39:33 2022] GET /identity/v3/users/cinder => generated 84 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:34.149557 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5735e5f6-afc9-4f82-b63a-305d10394572 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:34.171059 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b4f1a4-e08e-444f-971d-986043c96bfb None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:34.191074 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:34.204469 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:34.206883 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40ec6bcf-b5dc-4f02-9b1c-44134bbc660e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:34.207459 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ec6bcf-b5dc-4f02-9b1c-44134bbc660e None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:34.207735 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ec6bcf-b5dc-4f02-9b1c-44134bbc660e None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:34.208004 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ec6bcf-b5dc-4f02-9b1c-44134bbc660e None admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:34.208658 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40ec6bcf-b5dc-4f02-9b1c-44134bbc660e None admin] RBAC: Adding query filter params (name=service) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:34.208952 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40ec6bcf-b5dc-4f02-9b1c-44134bbc660e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:34.210314 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40ec6bcf-b5dc-4f02-9b1c-44134bbc660e None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:34.218578 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 258/506] 10.222.0.27 () {56 vars in 1103 bytes} [Sun Sep 11 00:39:34 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:34.219388 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40ec6bcf-b5dc-4f02-9b1c-44134bbc660e None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:34.253753 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5735e5f6-afc9-4f82-b63a-305d10394572 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:34.286529 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b15be34e-29d4-43c7-8899-24b0d382e688 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:34.287174 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15be34e-29d4-43c7-8899-24b0d382e688 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:34.287460 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15be34e-29d4-43c7-8899-24b0d382e688 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:34.287729 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15be34e-29d4-43c7-8899-24b0d382e688 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:34.296388 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b15be34e-29d4-43c7-8899-24b0d382e688 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:34.297837 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b15be34e-29d4-43c7-8899-24b0d382e688 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:34.309796 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-b15be34e-29d4-43c7-8899-24b0d382e688 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 11 00:39:34.309796 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:34.309796 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:34.309796 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:34.309796 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:34.309796 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:34.309796 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:34.309796 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:34.309796 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:34.309796 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:34.309796 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:34.309796 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:34.309796 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 11 00:39:34.309796 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 11 00:39:34.309796 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 11 00:39:34.309796 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 11 00:39:34.310754 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:34.310754 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:34.310754 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 11 00:39:34.310754 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:39:34.310754 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 11 00:39:34.310754 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:39:34.310754 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:34.310754 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:34.310754 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:34.310754 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:34.310754 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:34.310754 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:34.310754 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:34.310754 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:34.310754 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:34.310754 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:34.310754 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:34.310754 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:34.311707 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:34.311707 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:34.311707 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 11 00:39:34.311707 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 11 00:39:34.311707 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 11 00:39:34.311707 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 11 00:39:34.311707 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 11 00:39:34.311707 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 11 00:39:34.311707 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 11 00:39:34.311707 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:34.311707 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 249/507] 10.222.0.27 () {56 vars in 1102 bytes} [Sun Sep 11 00:39:34 2022] GET /identity/v3/users/neutron => generated 85 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:34.312281 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c4619b-5767-4f6d-899c-cb0c5868e6f9 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:34.312825 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c4619b-5767-4f6d-899c-cb0c5868e6f9 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:34.312825 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c4619b-5767-4f6d-899c-cb0c5868e6f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:34.319503 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40ec6bcf-b5dc-4f02-9b1c-44134bbc660e None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:34.330042 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-21c4619b-5767-4f6d-899c-cb0c5868e6f9 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:39:34.331257 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-21c4619b-5767-4f6d-899c-cb0c5868e6f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:39:34.354021 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfecc72a-2f62-4f91-8b6d-bb2b32b6640d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:34.354616 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfecc72a-2f62-4f91-8b6d-bb2b32b6640d None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:34.354924 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfecc72a-2f62-4f91-8b6d-bb2b32b6640d None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:34.355196 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfecc72a-2f62-4f91-8b6d-bb2b32b6640d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:34.395712 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 250/508] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:39:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:39:34.396533 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c4619b-5767-4f6d-899c-cb0c5868e6f9 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:34.405663 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:34.425895 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfecc72a-2f62-4f91-8b6d-bb2b32b6640d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:34.428007 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfecc72a-2f62-4f91-8b6d-bb2b32b6640d None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:34.449844 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:34.517409 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c4619b-5767-4f6d-899c-cb0c5868e6f9 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:34.533648 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 259/509] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:39:34 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 11 00:39:34.533648 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfecc72a-2f62-4f91-8b6d-bb2b32b6640d None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:34.564789 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2922430d-6d15-43ea-900e-e50b3bbfb7d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:34.565806 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2922430d-6d15-43ea-900e-e50b3bbfb7d9 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:34.565806 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2922430d-6d15-43ea-900e-e50b3bbfb7d9 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:34.566079 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2922430d-6d15-43ea-900e-e50b3bbfb7d9 None admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:34.566746 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2922430d-6d15-43ea-900e-e50b3bbfb7d9 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:34.567074 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2922430d-6d15-43ea-900e-e50b3bbfb7d9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:34.568538 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2922430d-6d15-43ea-900e-e50b3bbfb7d9 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:34.607657 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 251/510] 10.222.0.27 () {56 vars in 1137 bytes} [Sun Sep 11 00:39:34 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:34.608508 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2922430d-6d15-43ea-900e-e50b3bbfb7d9 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:34.611783 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:34.635810 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfecc72a-2f62-4f91-8b6d-bb2b32b6640d None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:34.674734 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b473d765-a440-49dc-95c0-71634d55968d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:34.675387 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b473d765-a440-49dc-95c0-71634d55968d None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:34.675660 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b473d765-a440-49dc-95c0-71634d55968d None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:34.675931 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b473d765-a440-49dc-95c0-71634d55968d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:34.686054 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b473d765-a440-49dc-95c0-71634d55968d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:34.687607 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b473d765-a440-49dc-95c0-71634d55968d None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:34.702624 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-b473d765-a440-49dc-95c0-71634d55968d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:34.702624 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:34.702624 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:34.702624 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:34.702624 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:34.702624 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:34.702624 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:34.702624 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:34.702624 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:34.702624 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:34.702624 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:34.702624 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:34.702624 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:39:34.702624 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:39:34.702624 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:39:34.702624 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:39:34.708199 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:34.708199 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:34.708199 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:34.708199 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:34.708199 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:34.708199 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:34.708199 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:34.708199 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:34.708199 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:34.708199 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:34.708199 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:34.708199 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:34.708199 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:34.708199 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:34.708199 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:34.708199 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:34.708199 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:39:34.709319 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:39:34.709319 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:34.709319 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:34.709319 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 260/511] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:39:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:34.709319 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b473d765-a440-49dc-95c0-71634d55968d None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:34.718851 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2922430d-6d15-43ea-900e-e50b3bbfb7d9 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:34.720487 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:34.753654 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9c7579-b547-41fe-b7a9-71cdc56c507e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:34.754253 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9c7579-b547-41fe-b7a9-71cdc56c507e None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:34.754530 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9c7579-b547-41fe-b7a9-71cdc56c507e None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:34.754818 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9c7579-b547-41fe-b7a9-71cdc56c507e None admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:34.755547 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af9c7579-b547-41fe-b7a9-71cdc56c507e None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:34.755846 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af9c7579-b547-41fe-b7a9-71cdc56c507e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:34.757292 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af9c7579-b547-41fe-b7a9-71cdc56c507e None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:34.775270 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 252/512] 10.222.0.27 () {56 vars in 1139 bytes} [Sun Sep 11 00:39:34 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:34.776070 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9c7579-b547-41fe-b7a9-71cdc56c507e None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:34.806473 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b473d765-a440-49dc-95c0-71634d55968d None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:34.839996 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a109fd34-8625-496b-bbc4-1376510f8da1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:34.840620 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a109fd34-8625-496b-bbc4-1376510f8da1 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:34.840925 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a109fd34-8625-496b-bbc4-1376510f8da1 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:34.841195 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a109fd34-8625-496b-bbc4-1376510f8da1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:34.843086 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:34.853657 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a109fd34-8625-496b-bbc4-1376510f8da1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:34.855147 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a109fd34-8625-496b-bbc4-1376510f8da1 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:34.869069 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-a109fd34-8625-496b-bbc4-1376510f8da1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:34.869069 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:34.869069 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:34.869069 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:34.869069 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:34.869069 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:34.869069 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:34.869069 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:34.869069 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:34.869069 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:34.869069 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:34.869069 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:34.869069 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:39:34.869069 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:39:34.869069 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:39:34.869069 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:39:34.870148 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:34.870148 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:34.870148 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:34.870148 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:34.870148 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:34.870148 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:34.870148 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:34.870148 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:34.870148 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:34.870148 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:34.870148 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:34.870148 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:34.870148 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:34.870148 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:34.870148 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:34.870148 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:34.870148 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:39:34.871233 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:39:34.871233 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:34.871233 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:34.871233 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 261/513] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:39:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:34.871233 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a109fd34-8625-496b-bbc4-1376510f8da1 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:34.875521 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9c7579-b547-41fe-b7a9-71cdc56c507e None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:34.909856 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b592e0a6-7f1d-4591-b6a7-1695299b87a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:34.910711 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b592e0a6-7f1d-4591-b6a7-1695299b87a3 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:34.911016 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b592e0a6-7f1d-4591-b6a7-1695299b87a3 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:34.911284 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b592e0a6-7f1d-4591-b6a7-1695299b87a3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:34.938237 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:34.970403 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a109fd34-8625-496b-bbc4-1376510f8da1 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:34.982012 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b592e0a6-7f1d-4591-b6a7-1695299b87a3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:34.983518 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b592e0a6-7f1d-4591-b6a7-1695299b87a3 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:35.003384 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14107b5e-dcf6-459e-816a-1d9a15394ca8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:35.004003 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14107b5e-dcf6-459e-816a-1d9a15394ca8 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:35.004275 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14107b5e-dcf6-459e-816a-1d9a15394ca8 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:35.004542 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14107b5e-dcf6-459e-816a-1d9a15394ca8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:35.015116 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14107b5e-dcf6-459e-816a-1d9a15394ca8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:35.016534 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14107b5e-dcf6-459e-816a-1d9a15394ca8 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:35.031301 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-14107b5e-dcf6-459e-816a-1d9a15394ca8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:35.031301 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:35.031301 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:35.031301 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:35.031301 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:35.031301 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:35.031301 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:35.031301 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:35.031301 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:35.031301 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:35.031301 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:35.031301 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:35.031301 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:39:35.031301 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:39:35.031301 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:39:35.031301 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:39:35.032374 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:35.032374 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:35.032374 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:35.032374 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:35.032374 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:35.032374 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:35.032374 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:35.032374 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:35.032374 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:35.032374 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:35.032374 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:35.032374 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:35.032374 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:35.032374 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:35.032374 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:35.032374 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:35.032374 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:39:35.033221 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:39:35.033221 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:35.033221 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:35.033221 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 262/514] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:39:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:35.033221 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14107b5e-dcf6-459e-816a-1d9a15394ca8 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:35.075656 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:35.088632 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 253/515] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:39:34 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 11 00:39:35.088632 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b592e0a6-7f1d-4591-b6a7-1695299b87a3 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:35.137861 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14107b5e-dcf6-459e-816a-1d9a15394ca8 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:35.148437 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:35.171478 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb92105-3bf5-4412-8f77-b7a24d2d2a32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:35.172056 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb92105-3bf5-4412-8f77-b7a24d2d2a32 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:35.172330 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb92105-3bf5-4412-8f77-b7a24d2d2a32 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:35.172595 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb92105-3bf5-4412-8f77-b7a24d2d2a32 None admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:35.173229 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fb92105-3bf5-4412-8f77-b7a24d2d2a32 None admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:35.173524 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fb92105-3bf5-4412-8f77-b7a24d2d2a32 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:35.174914 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fb92105-3bf5-4412-8f77-b7a24d2d2a32 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:35.187645 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 263/516] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:39:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:35.188440 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb92105-3bf5-4412-8f77-b7a24d2d2a32 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:35.188949 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b592e0a6-7f1d-4591-b6a7-1695299b87a3 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:35.225045 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e405b14-6677-44d4-a5f4-ee4d54de9bf5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:35.225549 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e405b14-6677-44d4-a5f4-ee4d54de9bf5 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:35.225820 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e405b14-6677-44d4-a5f4-ee4d54de9bf5 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:35.226082 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e405b14-6677-44d4-a5f4-ee4d54de9bf5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:35.283554 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:35.287872 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb92105-3bf5-4412-8f77-b7a24d2d2a32 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:35.295491 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e405b14-6677-44d4-a5f4-ee4d54de9bf5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:35.296966 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e405b14-6677-44d4-a5f4-ee4d54de9bf5 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:35.324338 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a98ce2d-1794-4d8d-b043-0f97b6ac21bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:35.325025 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a98ce2d-1794-4d8d-b043-0f97b6ac21bb None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:35.325376 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a98ce2d-1794-4d8d-b043-0f97b6ac21bb None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:35.325649 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a98ce2d-1794-4d8d-b043-0f97b6ac21bb None admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:35.326289 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a98ce2d-1794-4d8d-b043-0f97b6ac21bb None admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:35.326593 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a98ce2d-1794-4d8d-b043-0f97b6ac21bb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:35.327988 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a98ce2d-1794-4d8d-b043-0f97b6ac21bb None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:35.340091 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 264/517] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:39:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:35.341057 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a98ce2d-1794-4d8d-b043-0f97b6ac21bb None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:35.358521 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:35.400750 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 254/518] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:39:35 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 11 00:39:35.401578 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e405b14-6677-44d4-a5f4-ee4d54de9bf5 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:35.444580 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a98ce2d-1794-4d8d-b043-0f97b6ac21bb None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:35.479259 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b194a9e3-2ce7-4e7a-92c0-4f84966cf6bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:35.479782 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b194a9e3-2ce7-4e7a-92c0-4f84966cf6bb None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:35.480057 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b194a9e3-2ce7-4e7a-92c0-4f84966cf6bb None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:35.480326 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b194a9e3-2ce7-4e7a-92c0-4f84966cf6bb None admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:35.480965 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b194a9e3-2ce7-4e7a-92c0-4f84966cf6bb None admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:35.481262 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b194a9e3-2ce7-4e7a-92c0-4f84966cf6bb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:35.482618 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b194a9e3-2ce7-4e7a-92c0-4f84966cf6bb None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:35.494353 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 265/519] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:39:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:35.495197 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b194a9e3-2ce7-4e7a-92c0-4f84966cf6bb None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:35.498163 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e405b14-6677-44d4-a5f4-ee4d54de9bf5 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:35.514960 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:35.532750 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eac5406-1b0e-4159-bf81-36932e83f744 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:35.533377 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eac5406-1b0e-4159-bf81-36932e83f744 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:35.533654 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eac5406-1b0e-4159-bf81-36932e83f744 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:35.533928 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eac5406-1b0e-4159-bf81-36932e83f744 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:35.544233 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eac5406-1b0e-4159-bf81-36932e83f744 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:35.545695 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eac5406-1b0e-4159-bf81-36932e83f744 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:35.560671 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-9eac5406-1b0e-4159-bf81-36932e83f744 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:35.560671 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:35.560671 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:35.560671 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:35.560671 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:35.560671 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:35.560671 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:35.560671 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:35.560671 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:35.560671 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:35.560671 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:35.560671 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:35.560671 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:39:35.560671 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:39:35.560671 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:39:35.560671 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:39:35.561591 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:35.561591 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:35.561591 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:35.561591 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:35.561591 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:35.561591 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:35.561591 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:35.561591 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:35.561591 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:35.561591 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:35.561591 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:35.561591 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:35.561591 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:35.561591 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:35.561591 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:35.561591 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:35.561591 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:39:35.562457 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:39:35.562457 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:35.562457 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:35.562457 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 255/520] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:39:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:35.562671 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eac5406-1b0e-4159-bf81-36932e83f744 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:35.580776 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:35.597457 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b194a9e3-2ce7-4e7a-92c0-4f84966cf6bb None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:35.630274 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbce2e15-c687-4f4d-9499-3419bd075ded None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:35.630921 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbce2e15-c687-4f4d-9499-3419bd075ded None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:35.631198 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbce2e15-c687-4f4d-9499-3419bd075ded None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:35.631462 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbce2e15-c687-4f4d-9499-3419bd075ded None admin] PATH_INFO: `/v3/users/glance` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:35.638857 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbce2e15-c687-4f4d-9499-3419bd075ded None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:35.640325 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbce2e15-c687-4f4d-9499-3419bd075ded None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:35.652109 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-bbce2e15-c687-4f4d-9499-3419bd075ded None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 11 00:39:35.652109 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:35.652109 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:35.652109 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:35.652109 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:35.652109 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:35.652109 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:35.652109 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:35.652109 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:35.652109 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:35.652109 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:35.652109 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:35.652109 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 11 00:39:35.652109 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 11 00:39:35.652109 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 11 00:39:35.652109 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 11 00:39:35.653201 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:35.653201 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:35.653201 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 11 00:39:35.653201 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:39:35.653201 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 11 00:39:35.653201 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:39:35.653201 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:35.653201 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:35.653201 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:35.653201 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:35.653201 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:35.653201 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:35.653201 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:35.653201 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:35.653201 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:35.653201 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:35.653201 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:35.653201 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:35.654423 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:35.654423 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:35.654423 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 11 00:39:35.654423 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 11 00:39:35.654423 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 11 00:39:35.654423 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 11 00:39:35.654423 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 11 00:39:35.654423 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 11 00:39:35.654423 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 11 00:39:35.654423 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:35.654423 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 266/521] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:39:35 2022] GET /identity/v3/users/glance => generated 84 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:35.654423 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbce2e15-c687-4f4d-9499-3419bd075ded None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:35.673953 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eac5406-1b0e-4159-bf81-36932e83f744 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:35.708306 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109456e5-6c95-4013-b7c3-1935dce1d3c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:35.708921 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109456e5-6c95-4013-b7c3-1935dce1d3c2 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:35.709198 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109456e5-6c95-4013-b7c3-1935dce1d3c2 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:35.709466 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109456e5-6c95-4013-b7c3-1935dce1d3c2 None admin] PATH_INFO: `/v3/projects/service` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:35.716130 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-109456e5-6c95-4013-b7c3-1935dce1d3c2 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:35.717583 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-109456e5-6c95-4013-b7c3-1935dce1d3c2 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:35.729795 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-109456e5-6c95-4013-b7c3-1935dce1d3c2 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:39:35.729795 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:35.729795 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:35.729795 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:35.729795 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:35.729795 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:35.729795 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:35.729795 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:35.729795 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:35.729795 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:35.729795 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:35.729795 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:35.729795 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 11 00:39:35.729795 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 11 00:39:35.729795 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 11 00:39:35.730820 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 11 00:39:35.730820 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:35.730820 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:35.730820 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:35.730820 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:35.730820 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:35.730820 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:35.730820 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:35.730820 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:35.730820 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:35.730820 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:35.730820 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:35.730820 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:35.730820 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:35.730820 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:35.730820 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:35.730820 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:35.731722 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 11 00:39:35.731722 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 11 00:39:35.731722 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 11 00:39:35.731722 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 11 00:39:35.731722 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 11 00:39:35.731722 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 11 00:39:35.731722 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:39:35.731722 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:35.732137 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:35.732210 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 256/522] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:39:35 2022] GET /identity/v3/projects/service => generated 88 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:35.732210 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109456e5-6c95-4013-b7c3-1935dce1d3c2 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:35.754159 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbce2e15-c687-4f4d-9499-3419bd075ded None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:35.787895 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-902232a8-df3e-4c32-8f74-7b763e808c58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:35.788524 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902232a8-df3e-4c32-8f74-7b763e808c58 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:35.788835 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902232a8-df3e-4c32-8f74-7b763e808c58 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:35.789104 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902232a8-df3e-4c32-8f74-7b763e808c58 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:35.799690 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-902232a8-df3e-4c32-8f74-7b763e808c58 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:35.801171 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-902232a8-df3e-4c32-8f74-7b763e808c58 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:35.809015 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:35.815995 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-902232a8-df3e-4c32-8f74-7b763e808c58 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:35.815995 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:35.815995 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:35.815995 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:35.815995 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:35.815995 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:35.815995 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:35.815995 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:35.815995 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:35.815995 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:35.815995 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:35.815995 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:35.815995 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:39:35.815995 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:39:35.815995 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:39:35.815995 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:39:35.816942 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:35.816942 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:35.816942 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:35.816942 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:35.816942 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:35.816942 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:35.816942 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:35.816942 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:35.816942 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:35.816942 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:35.816942 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:35.816942 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:35.816942 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:35.816942 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:35.816942 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:35.816942 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:35.816942 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:39:35.818312 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:39:35.818312 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:35.818312 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:35.818312 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 267/523] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:39:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:35.818312 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-902232a8-df3e-4c32-8f74-7b763e808c58 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:35.851621 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109456e5-6c95-4013-b7c3-1935dce1d3c2 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:35.887635 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629ff2a4-d5c4-4cb5-8393-a9eb7f3179aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:35.888227 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629ff2a4-d5c4-4cb5-8393-a9eb7f3179aa None admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:35.888502 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629ff2a4-d5c4-4cb5-8393-a9eb7f3179aa None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:35.888766 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629ff2a4-d5c4-4cb5-8393-a9eb7f3179aa None admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:35.889395 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-629ff2a4-d5c4-4cb5-8393-a9eb7f3179aa None admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:35.890855 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-629ff2a4-d5c4-4cb5-8393-a9eb7f3179aa None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:35.911443 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 257/524] 10.222.0.27 () {60 vars in 1132 bytes} [Sun Sep 11 00:39:35 2022] POST /identity/v3/users => generated 247 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:39:35.919930 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629ff2a4-d5c4-4cb5-8393-a9eb7f3179aa None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:35.927318 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-902232a8-df3e-4c32-8f74-7b763e808c58 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:35.962081 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:35.969258 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc6f2b3-7928-499c-a370-fc1a855effab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:35.969880 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc6f2b3-7928-499c-a370-fc1a855effab None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:35.970156 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc6f2b3-7928-499c-a370-fc1a855effab None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:35.970424 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc6f2b3-7928-499c-a370-fc1a855effab None admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:35.971127 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fc6f2b3-7928-499c-a370-fc1a855effab None admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:35.971426 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fc6f2b3-7928-499c-a370-fc1a855effab None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:35.972845 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fc6f2b3-7928-499c-a370-fc1a855effab None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:35.992794 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 268/525] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:39:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:35.992794 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc6f2b3-7928-499c-a370-fc1a855effab None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:36.045183 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629ff2a4-d5c4-4cb5-8393-a9eb7f3179aa None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:36.084777 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:36.092648 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a052c0f-70d4-4db0-a9e7-a8885ce11f3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:36.093283 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a052c0f-70d4-4db0-a9e7-a8885ce11f3b None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:36.093553 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a052c0f-70d4-4db0-a9e7-a8885ce11f3b None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:36.093823 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a052c0f-70d4-4db0-a9e7-a8885ce11f3b None admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:36.094994 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a052c0f-70d4-4db0-a9e7-a8885ce11f3b None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:36.094994 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a052c0f-70d4-4db0-a9e7-a8885ce11f3b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:36.096294 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a052c0f-70d4-4db0-a9e7-a8885ce11f3b None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:36.116105 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 258/526] 10.222.0.27 () {56 vars in 1137 bytes} [Sun Sep 11 00:39:35 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:36.117678 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a052c0f-70d4-4db0-a9e7-a8885ce11f3b None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:36.122745 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc6f2b3-7928-499c-a370-fc1a855effab None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:36.162358 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc11c680-53a1-4264-b7e5-918ef8dcdcfd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:36.163000 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc11c680-53a1-4264-b7e5-918ef8dcdcfd None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:36.163279 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc11c680-53a1-4264-b7e5-918ef8dcdcfd None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:36.163547 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc11c680-53a1-4264-b7e5-918ef8dcdcfd None admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:36.164768 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc11c680-53a1-4264-b7e5-918ef8dcdcfd None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:36.165076 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc11c680-53a1-4264-b7e5-918ef8dcdcfd None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:36.166512 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc11c680-53a1-4264-b7e5-918ef8dcdcfd None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:36.189046 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 269/527] 10.222.0.27 () {56 vars in 1148 bytes} [Sun Sep 11 00:39:35 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:36.189046 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc11c680-53a1-4264-b7e5-918ef8dcdcfd None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:36.242060 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a052c0f-70d4-4db0-a9e7-a8885ce11f3b None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:36.243158 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:36.284844 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4fa1e24-4b64-4f80-a68b-e85e54a7770d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:36.285717 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4fa1e24-4b64-4f80-a68b-e85e54a7770d None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:36.285993 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4fa1e24-4b64-4f80-a68b-e85e54a7770d None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:36.286266 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4fa1e24-4b64-4f80-a68b-e85e54a7770d None admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:36.286971 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4fa1e24-4b64-4f80-a68b-e85e54a7770d None admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:36.287273 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4fa1e24-4b64-4f80-a68b-e85e54a7770d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:36.288637 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4fa1e24-4b64-4f80-a68b-e85e54a7770d None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:36.300988 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 259/528] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:39:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:36.301789 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4fa1e24-4b64-4f80-a68b-e85e54a7770d None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:36.302323 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc11c680-53a1-4264-b7e5-918ef8dcdcfd None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:36.341291 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e30fe7-b332-4c74-a7a2-a43499a9bd9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:36.341932 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e30fe7-b332-4c74-a7a2-a43499a9bd9f None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:36.342214 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e30fe7-b332-4c74-a7a2-a43499a9bd9f None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:36.342485 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e30fe7-b332-4c74-a7a2-a43499a9bd9f None admin] PATH_INFO: `/v3/projects/service` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:36.349365 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59e30fe7-b332-4c74-a7a2-a43499a9bd9f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:36.351022 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59e30fe7-b332-4c74-a7a2-a43499a9bd9f None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:36.353963 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:36.363992 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-59e30fe7-b332-4c74-a7a2-a43499a9bd9f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:39:36.363992 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:36.363992 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:36.363992 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:36.363992 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:36.363992 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:36.363992 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:36.363992 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:36.363992 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:36.363992 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:36.363992 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:36.363992 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:36.363992 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 11 00:39:36.363992 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 11 00:39:36.363992 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 11 00:39:36.364882 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 11 00:39:36.364882 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:36.364882 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:36.364882 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:36.364882 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:36.364882 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:36.364882 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:36.364882 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:36.364882 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:36.364882 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:36.364882 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:36.364882 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:36.364882 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:36.364882 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:36.364882 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:36.364882 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:36.364882 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:36.365719 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 11 00:39:36.365719 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 11 00:39:36.365719 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 11 00:39:36.365719 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 11 00:39:36.365719 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 11 00:39:36.365719 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 11 00:39:36.365719 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:39:36.365719 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:36.365719 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 270/529] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:39:36 2022] GET /identity/v3/projects/service => generated 88 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:36.366183 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e30fe7-b332-4c74-a7a2-a43499a9bd9f None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:36.419750 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4fa1e24-4b64-4f80-a68b-e85e54a7770d None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:36.455686 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b47ebe-28d9-4eaa-8164-7835ca4537ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:36.456258 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b47ebe-28d9-4eaa-8164-7835ca4537ab None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:36.456573 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b47ebe-28d9-4eaa-8164-7835ca4537ab None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:36.456929 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b47ebe-28d9-4eaa-8164-7835ca4537ab None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:36.484052 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e30fe7-b332-4c74-a7a2-a43499a9bd9f None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:36.494772 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:36.519787 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-794225dc-aab1-4783-9c0a-e6ce54b0c1e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:36.520399 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794225dc-aab1-4783-9c0a-e6ce54b0c1e4 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:36.520681 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794225dc-aab1-4783-9c0a-e6ce54b0c1e4 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:36.520957 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794225dc-aab1-4783-9c0a-e6ce54b0c1e4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:36.521684 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-794225dc-aab1-4783-9c0a-e6ce54b0c1e4 None admin] RBAC: Adding query filter params (role.id=0390f926e7ef4715ae18697033bde90e, scope.project.id=b504a5058bc6474c8cc077dfeec16799, user.id=835def0e6c6d4365a2767051509eb0d6) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:36.521992 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-794225dc-aab1-4783-9c0a-e6ce54b0c1e4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:36.524113 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-794225dc-aab1-4783-9c0a-e6ce54b0c1e4 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:36.529457 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 271/530] 10.222.0.27 () {56 vars in 1374 bytes} [Sun Sep 11 00:39:36 2022] GET /identity/v3/role_assignments?user.id=835def0e6c6d4365a2767051509eb0d6&scope.project.id=b504a5058bc6474c8cc077dfeec16799&role.id=0390f926e7ef4715ae18697033bde90e => generated 259 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:36.530247 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-794225dc-aab1-4783-9c0a-e6ce54b0c1e4 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:36.540639 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b47ebe-28d9-4eaa-8164-7835ca4537ab None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:36.542729 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b47ebe-28d9-4eaa-8164-7835ca4537ab None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:36.604339 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:36.662278 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-794225dc-aab1-4783-9c0a-e6ce54b0c1e4 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:36.684351 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 260/531] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:39:36 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 381 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 11 00:39:36.684351 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b47ebe-28d9-4eaa-8164-7835ca4537ab None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:36.710239 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92506a9a-0d6c-4c89-91dd-5958da6b326c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:36.711423 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92506a9a-0d6c-4c89-91dd-5958da6b326c None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:36.711977 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92506a9a-0d6c-4c89-91dd-5958da6b326c None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:36.712488 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92506a9a-0d6c-4c89-91dd-5958da6b326c None admin] PATH_INFO: `/v3/projects/service` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:36.721014 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92506a9a-0d6c-4c89-91dd-5958da6b326c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:36.723069 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92506a9a-0d6c-4c89-91dd-5958da6b326c None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:36.736663 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-92506a9a-0d6c-4c89-91dd-5958da6b326c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:39:36.736663 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:36.736663 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:36.736663 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:36.736663 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:36.736663 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:36.736663 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:36.736663 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:36.736663 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:36.736663 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:36.736663 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:36.736663 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:36.736663 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 11 00:39:36.736663 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 11 00:39:36.736663 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 11 00:39:36.737621 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 11 00:39:36.737621 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:36.737621 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:36.737621 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:36.737621 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:36.737621 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:36.737621 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:36.737621 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:36.737621 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:36.737621 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:36.737621 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:36.737621 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:36.737621 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:36.737621 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:36.737621 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:36.737621 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:36.737621 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:36.738476 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 11 00:39:36.738476 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 11 00:39:36.738476 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 11 00:39:36.738476 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 11 00:39:36.738476 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 11 00:39:36.738476 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 11 00:39:36.738476 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:39:36.738476 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:36.746838 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 272/532] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:39:36 2022] GET /identity/v3/projects/service => generated 88 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:36.746838 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92506a9a-0d6c-4c89-91dd-5958da6b326c None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:36.782926 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:36.803904 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b47ebe-28d9-4eaa-8164-7835ca4537ab None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:36.843009 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a676cd34-973d-42fe-a82f-5d4ba3734d4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:36.843638 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a676cd34-973d-42fe-a82f-5d4ba3734d4e None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:36.843919 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a676cd34-973d-42fe-a82f-5d4ba3734d4e None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:36.844191 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a676cd34-973d-42fe-a82f-5d4ba3734d4e None admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:36.844908 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a676cd34-973d-42fe-a82f-5d4ba3734d4e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:36.845217 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a676cd34-973d-42fe-a82f-5d4ba3734d4e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:36.846824 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a676cd34-973d-42fe-a82f-5d4ba3734d4e None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:36.864873 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92506a9a-0d6c-4c89-91dd-5958da6b326c None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:36.871607 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 261/533] 10.222.0.27 () {56 vars in 1148 bytes} [Sun Sep 11 00:39:36 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:36.871607 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a676cd34-973d-42fe-a82f-5d4ba3734d4e None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:36.880851 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:36.915902 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73cec34-9315-441d-b326-1d604152e56d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:36.916780 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73cec34-9315-441d-b326-1d604152e56d None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:36.917207 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73cec34-9315-441d-b326-1d604152e56d None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:36.918084 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73cec34-9315-441d-b326-1d604152e56d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:36.934195 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f73cec34-9315-441d-b326-1d604152e56d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:36.936656 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f73cec34-9315-441d-b326-1d604152e56d None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:36.956874 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-f73cec34-9315-441d-b326-1d604152e56d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:36.956874 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:36.956874 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:36.956874 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:36.956874 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:36.956874 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:36.956874 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:36.956874 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:36.956874 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:36.956874 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:36.956874 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:36.956874 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:36.956874 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:39:36.956874 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:39:36.956874 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:39:36.956874 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:39:36.958006 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:36.958006 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:36.958006 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:36.958006 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:36.958006 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:36.958006 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:36.958006 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:36.958006 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:36.958006 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:36.958006 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:36.958006 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:36.958006 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:36.958006 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:36.958006 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:36.958006 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:36.958006 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:36.958006 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:39:36.959069 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:39:36.959069 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:36.959069 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:36.961665 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 273/534] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:39:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:36.962947 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73cec34-9315-441d-b326-1d604152e56d None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:36.987220 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a676cd34-973d-42fe-a82f-5d4ba3734d4e None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:37.027496 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf24142-ce13-4ab8-9eaa-29e1334dcdaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:37.028314 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf24142-ce13-4ab8-9eaa-29e1334dcdaf None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:37.028591 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf24142-ce13-4ab8-9eaa-29e1334dcdaf None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:37.028863 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf24142-ce13-4ab8-9eaa-29e1334dcdaf None admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:37.029717 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdf24142-ce13-4ab8-9eaa-29e1334dcdaf None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:37.030344 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdf24142-ce13-4ab8-9eaa-29e1334dcdaf None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:37.032255 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdf24142-ce13-4ab8-9eaa-29e1334dcdaf None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:37.052083 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 262/535] 10.222.0.27 () {56 vars in 1148 bytes} [Sun Sep 11 00:39:36 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:37.053020 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf24142-ce13-4ab8-9eaa-29e1334dcdaf None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:37.071156 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:37.088870 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73cec34-9315-441d-b326-1d604152e56d None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:37.138758 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-958b13f5-5f0e-4512-b5e2-16ad2acc6d1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:37.139613 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-958b13f5-5f0e-4512-b5e2-16ad2acc6d1c None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:37.139904 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-958b13f5-5f0e-4512-b5e2-16ad2acc6d1c None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:37.140177 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-958b13f5-5f0e-4512-b5e2-16ad2acc6d1c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:37.141396 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-958b13f5-5f0e-4512-b5e2-16ad2acc6d1c None admin] RBAC: Adding query filter params (role.id=0390f926e7ef4715ae18697033bde90e, scope.project.id=b504a5058bc6474c8cc077dfeec16799, user.id=22374eaff0834245b88506dcd7b24ede) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:37.141709 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-958b13f5-5f0e-4512-b5e2-16ad2acc6d1c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:37.143594 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-958b13f5-5f0e-4512-b5e2-16ad2acc6d1c None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:37.154907 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 274/536] 10.222.0.27 () {56 vars in 1374 bytes} [Sun Sep 11 00:39:36 2022] GET /identity/v3/role_assignments?user.id=22374eaff0834245b88506dcd7b24ede&scope.project.id=b504a5058bc6474c8cc077dfeec16799&role.id=0390f926e7ef4715ae18697033bde90e => generated 259 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:37.154907 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-958b13f5-5f0e-4512-b5e2-16ad2acc6d1c None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:37.169708 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:37.242018 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf24142-ce13-4ab8-9eaa-29e1334dcdaf None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:37.318005 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb69fd2-50a2-4e46-b617-b440ba0bebad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:37.318537 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb69fd2-50a2-4e46-b617-b440ba0bebad None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:37.318821 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb69fd2-50a2-4e46-b617-b440ba0bebad None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:37.319095 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb69fd2-50a2-4e46-b617-b440ba0bebad None admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:37.319751 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feb69fd2-50a2-4e46-b617-b440ba0bebad None admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:37.320633 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feb69fd2-50a2-4e46-b617-b440ba0bebad None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:37.322056 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feb69fd2-50a2-4e46-b617-b440ba0bebad None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:37.335008 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 263/537] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:39:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 282 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:37.341179 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb69fd2-50a2-4e46-b617-b440ba0bebad None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:37.375050 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-958b13f5-5f0e-4512-b5e2-16ad2acc6d1c None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:37.423546 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b61561-3e56-45d5-8f10-05de0f199e9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:37.424498 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b61561-3e56-45d5-8f10-05de0f199e9f None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:37.425031 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b61561-3e56-45d5-8f10-05de0f199e9f None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:37.425505 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b61561-3e56-45d5-8f10-05de0f199e9f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:37.426479 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63b61561-3e56-45d5-8f10-05de0f199e9f None admin] RBAC: Adding query filter params (role.id=0390f926e7ef4715ae18697033bde90e, scope.project.id=b504a5058bc6474c8cc077dfeec16799, user.id=a5f7b13283cb4ebfb25160aa6ad0e2f4) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:37.427031 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63b61561-3e56-45d5-8f10-05de0f199e9f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:37.430362 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63b61561-3e56-45d5-8f10-05de0f199e9f None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:37.437882 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 275/538] 10.222.0.27 () {56 vars in 1374 bytes} [Sun Sep 11 00:39:37 2022] GET /identity/v3/role_assignments?user.id=a5f7b13283cb4ebfb25160aa6ad0e2f4&scope.project.id=b504a5058bc6474c8cc077dfeec16799&role.id=0390f926e7ef4715ae18697033bde90e => generated 259 bytes in 286 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:37.480907 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb69fd2-50a2-4e46-b617-b440ba0bebad None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:37.527704 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:37.543099 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe57afa-b8c5-4ba7-a872-f5439a235ab3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:37.543750 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe57afa-b8c5-4ba7-a872-f5439a235ab3 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:37.544026 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe57afa-b8c5-4ba7-a872-f5439a235ab3 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:37.544295 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe57afa-b8c5-4ba7-a872-f5439a235ab3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:37.555716 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffe57afa-b8c5-4ba7-a872-f5439a235ab3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:37.557292 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffe57afa-b8c5-4ba7-a872-f5439a235ab3 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:37.575876 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-ffe57afa-b8c5-4ba7-a872-f5439a235ab3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:39:37.575876 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:37.575876 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:37.575876 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:37.575876 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:37.575876 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:37.575876 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:37.575876 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:37.575876 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:37.575876 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:37.575876 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:37.575876 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:37.575876 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 11 00:39:37.575876 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 11 00:39:37.575876 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 11 00:39:37.576778 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 11 00:39:37.576778 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:37.576778 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:37.576778 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:37.576778 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:37.576778 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:37.576778 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:37.576778 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:37.576778 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:37.576778 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:37.576778 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:37.576778 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:37.576778 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:37.576778 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:37.576778 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:37.576778 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:37.576778 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:37.577629 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 11 00:39:37.577629 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 11 00:39:37.577629 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 11 00:39:37.577629 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 11 00:39:37.577629 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 11 00:39:37.577629 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 11 00:39:37.577629 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:39:37.577629 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:37.577629 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 264/539] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:39:37 2022] GET /identity/v3/projects/service => generated 88 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:39:37.582980 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b61561-3e56-45d5-8f10-05de0f199e9f None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:37.603733 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:37.700468 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b61561-3e56-45d5-8f10-05de0f199e9f None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:37.738022 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-820c3cf5-8265-4743-a6a0-7d7c7b7af143 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:37.739038 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-820c3cf5-8265-4743-a6a0-7d7c7b7af143 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:37.739523 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-820c3cf5-8265-4743-a6a0-7d7c7b7af143 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:37.740014 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-820c3cf5-8265-4743-a6a0-7d7c7b7af143 None admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:37.740926 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-820c3cf5-8265-4743-a6a0-7d7c7b7af143 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:37.741441 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-820c3cf5-8265-4743-a6a0-7d7c7b7af143 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:37.743471 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-820c3cf5-8265-4743-a6a0-7d7c7b7af143 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:37.756848 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 276/540] 10.222.0.27 () {56 vars in 1148 bytes} [Sun Sep 11 00:39:37 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:37.763217 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe57afa-b8c5-4ba7-a872-f5439a235ab3 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:37.876997 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe57afa-b8c5-4ba7-a872-f5439a235ab3 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:37.914397 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887e701b-8468-4314-b42d-7f212571d2da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:37.915398 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887e701b-8468-4314-b42d-7f212571d2da None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:37.915901 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887e701b-8468-4314-b42d-7f212571d2da None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:37.916374 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887e701b-8468-4314-b42d-7f212571d2da None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:37.917325 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-887e701b-8468-4314-b42d-7f212571d2da None admin] RBAC: Adding query filter params (role.id=0390f926e7ef4715ae18697033bde90e, scope.project.id=b504a5058bc6474c8cc077dfeec16799, user.id=06b1c911731f4267a5f37e91d948e7e9) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:37.917848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-887e701b-8468-4314-b42d-7f212571d2da None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:37.919527 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-887e701b-8468-4314-b42d-7f212571d2da None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:37.930028 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 265/541] 10.222.0.27 () {56 vars in 1374 bytes} [Sun Sep 11 00:39:37 2022] GET /identity/v3/role_assignments?user.id=06b1c911731f4267a5f37e91d948e7e9&scope.project.id=b504a5058bc6474c8cc077dfeec16799&role.id=0390f926e7ef4715ae18697033bde90e => generated 259 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:39:38.270142 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd98943c-c038-4bc6-baa0-c6e1c4e4bac1 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:38.271208 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd98943c-c038-4bc6-baa0-c6e1c4e4bac1 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:38.271702 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd98943c-c038-4bc6-baa0-c6e1c4e4bac1 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:38.273584 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 277/542] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:39:38 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:39:38.279696 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7bd7ed2-1b71-4b3a-883e-06576a43fa33 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:38.280678 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7bd7ed2-1b71-4b3a-883e-06576a43fa33 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:38.281161 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7bd7ed2-1b71-4b3a-883e-06576a43fa33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:38.301720 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-f7bd7ed2-1b71-4b3a-883e-06576a43fa33 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:39:38.303268 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-f7bd7ed2-1b71-4b3a-883e-06576a43fa33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:39:38.355364 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:38.378726 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 266/543] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:39:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:39:38.654706 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd98943c-c038-4bc6-baa0-c6e1c4e4bac1 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:38.737320 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:38.768172 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd98943c-c038-4bc6-baa0-c6e1c4e4bac1 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:38.804830 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10713da-35cb-482e-97e8-65fe0865bd99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:38.805798 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10713da-35cb-482e-97e8-65fe0865bd99 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:38.806305 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10713da-35cb-482e-97e8-65fe0865bd99 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:38.806777 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10713da-35cb-482e-97e8-65fe0865bd99 None admin] PATH_INFO: `/v3/roles/service` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:38.817547 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d10713da-35cb-482e-97e8-65fe0865bd99 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:39:38.822768 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-d10713da-35cb-482e-97e8-65fe0865bd99 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 11 00:39:38.822768 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:38.822768 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:38.822768 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:38.822768 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:38.822768 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:38.822768 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:38.822768 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:38.822768 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:38.822768 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:38.822768 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:38.822768 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:38.822768 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:39:38.822768 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:39:38.822768 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:39:38.822768 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:39:38.823682 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:39:38.823682 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:39:38.823682 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:39:38.823682 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:39:38.823682 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:38.823682 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:38.823682 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:38.823682 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:38.823682 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:38.823682 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:38.823682 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:38.823682 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:38.823682 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:38.823682 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:38.823682 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:38.823682 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:38.824476 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:38.824476 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:38.824476 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:38.824476 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:38.824476 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:39:38.824476 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:39:38.824476 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:39:38.824476 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:39:38.824476 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:39:38.824476 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:39:38.824476 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 11 00:39:38.824476 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:38.831087 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 278/544] 10.222.0.27 () {56 vars in 1102 bytes} [Sun Sep 11 00:39:38 2022] GET /identity/v3/roles/service => generated 85 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:38.833031 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7bd7ed2-1b71-4b3a-883e-06576a43fa33 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:38.937565 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7bd7ed2-1b71-4b3a-883e-06576a43fa33 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:38.973622 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cca5bc3-9eba-430c-87e1-26beace48ff1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:38.974574 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cca5bc3-9eba-430c-87e1-26beace48ff1 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:38.975113 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cca5bc3-9eba-430c-87e1-26beace48ff1 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:38.975588 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cca5bc3-9eba-430c-87e1-26beace48ff1 None admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:38.976478 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cca5bc3-9eba-430c-87e1-26beace48ff1 None admin] RBAC: Adding query filter params (name=service) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:38.977051 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cca5bc3-9eba-430c-87e1-26beace48ff1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:38.978663 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cca5bc3-9eba-430c-87e1-26beace48ff1 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:38.987711 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 267/545] 10.222.0.27 () {56 vars in 1103 bytes} [Sun Sep 11 00:39:38 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:38.994335 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10713da-35cb-482e-97e8-65fe0865bd99 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:39.110594 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10713da-35cb-482e-97e8-65fe0865bd99 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:39.149301 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-836aa4bd-3e2f-4587-8be3-339eb672a36d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:39.150444 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836aa4bd-3e2f-4587-8be3-339eb672a36d None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:39.150967 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836aa4bd-3e2f-4587-8be3-339eb672a36d None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:39.151444 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836aa4bd-3e2f-4587-8be3-339eb672a36d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:39.159319 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:39.164373 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-836aa4bd-3e2f-4587-8be3-339eb672a36d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:39.166142 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-836aa4bd-3e2f-4587-8be3-339eb672a36d None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:39.182281 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-836aa4bd-3e2f-4587-8be3-339eb672a36d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:39.182281 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:39.182281 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:39.182281 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:39.182281 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:39.182281 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:39.182281 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:39.182281 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:39.182281 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:39.182281 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:39.182281 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:39.182281 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:39.182281 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:39:39.182281 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:39:39.182281 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:39:39.182281 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:39:39.183237 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:39.183237 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:39.183237 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:39.183237 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:39.183237 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:39.183237 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:39.183237 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:39.183237 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:39.183237 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:39.183237 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:39.183237 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:39.183237 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:39.183237 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:39.183237 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:39.183237 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:39.183237 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:39.183237 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:39:39.184108 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:39:39.184108 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:39.184108 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:39.186208 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 279/546] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:39:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:39.192054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cca5bc3-9eba-430c-87e1-26beace48ff1 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:39.240118 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:39.271007 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a87ada-8645-4092-858e-ae1326aa16f6 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:39.272002 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a87ada-8645-4092-858e-ae1326aa16f6 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:39.272500 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a87ada-8645-4092-858e-ae1326aa16f6 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:39.274984 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 280/547] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:39:39 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 11 00:39:39.280174 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4718235-2a1e-4765-ac6d-1bb9862fe019 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:39.280863 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4718235-2a1e-4765-ac6d-1bb9862fe019 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:39.282178 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4718235-2a1e-4765-ac6d-1bb9862fe019 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:39.306038 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-e4718235-2a1e-4765-ac6d-1bb9862fe019 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:39:39.307678 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-e4718235-2a1e-4765-ac6d-1bb9862fe019 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:39:39.357015 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cca5bc3-9eba-430c-87e1-26beace48ff1 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:39.385429 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 281/548] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:39:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:39:39.410492 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e3f961-5eba-4b1a-81ce-9af3eb975a61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:39.411153 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e3f961-5eba-4b1a-81ce-9af3eb975a61 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:39.411433 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e3f961-5eba-4b1a-81ce-9af3eb975a61 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:39.411709 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e3f961-5eba-4b1a-81ce-9af3eb975a61 None admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:39.412352 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46e3f961-5eba-4b1a-81ce-9af3eb975a61 None admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:39.412702 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46e3f961-5eba-4b1a-81ce-9af3eb975a61 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:39.414182 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46e3f961-5eba-4b1a-81ce-9af3eb975a61 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:39.426432 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 268/549] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:39:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:39.431859 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4718235-2a1e-4765-ac6d-1bb9862fe019 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:39.573779 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4718235-2a1e-4765-ac6d-1bb9862fe019 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:39.591629 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:39.610076 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dce7af2-c86c-41ea-aeb6-970c702ccf9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:39.610969 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dce7af2-c86c-41ea-aeb6-970c702ccf9f None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:39.611413 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dce7af2-c86c-41ea-aeb6-970c702ccf9f None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:39.611841 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dce7af2-c86c-41ea-aeb6-970c702ccf9f None admin] PATH_INFO: `/v3/users/swift` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:39.620369 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dce7af2-c86c-41ea-aeb6-970c702ccf9f None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:39.622009 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dce7af2-c86c-41ea-aeb6-970c702ccf9f None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:39.634922 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-8dce7af2-c86c-41ea-aeb6-970c702ccf9f None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 11 00:39:39.634922 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:39.634922 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:39.634922 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:39.634922 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:39.634922 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:39.634922 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:39.634922 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:39.634922 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:39.634922 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:39.634922 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:39.634922 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:39.634922 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 11 00:39:39.634922 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 11 00:39:39.634922 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 11 00:39:39.634922 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 11 00:39:39.636309 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:39.636309 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:39.636309 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 11 00:39:39.636309 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:39:39.636309 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 11 00:39:39.636309 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:39:39.636309 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:39.636309 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:39.636309 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:39.636309 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:39.636309 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:39.636309 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:39.636309 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:39.636309 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:39.636309 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:39.636309 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:39.636309 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:39.636309 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:39.637453 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:39.637453 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:39.637453 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 11 00:39:39.637453 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 11 00:39:39.637453 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 11 00:39:39.637453 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 11 00:39:39.637453 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 11 00:39:39.637453 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 11 00:39:39.637453 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 11 00:39:39.637453 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:39.639629 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 282/550] 10.222.0.27 () {56 vars in 1096 bytes} [Sun Sep 11 00:39:39 2022] GET /identity/v3/users/swift => generated 83 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:39.645532 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e3f961-5eba-4b1a-81ce-9af3eb975a61 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:39.742635 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dce7af2-c86c-41ea-aeb6-970c702ccf9f None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:39.835271 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e3f961-5eba-4b1a-81ce-9af3eb975a61 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:39.837000 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:39.881271 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5bae3d-4da3-49b4-ac45-3626ead15c73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:39.881880 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5bae3d-4da3-49b4-ac45-3626ead15c73 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:39.882162 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5bae3d-4da3-49b4-ac45-3626ead15c73 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:39.883016 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5bae3d-4da3-49b4-ac45-3626ead15c73 None admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:39.883756 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be5bae3d-4da3-49b4-ac45-3626ead15c73 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:39.884060 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be5bae3d-4da3-49b4-ac45-3626ead15c73 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:39.885511 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be5bae3d-4da3-49b4-ac45-3626ead15c73 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:39.904507 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 269/551] 10.222.0.27 () {56 vars in 1135 bytes} [Sun Sep 11 00:39:39 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:39.911707 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5bae3d-4da3-49b4-ac45-3626ead15c73 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:39.938338 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dce7af2-c86c-41ea-aeb6-970c702ccf9f None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:40.001940 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a25f25-f15a-4db2-811d-8b95bb10c5ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:40.002577 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a25f25-f15a-4db2-811d-8b95bb10c5ce None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:40.002894 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a25f25-f15a-4db2-811d-8b95bb10c5ce None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:40.003175 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a25f25-f15a-4db2-811d-8b95bb10c5ce None admin] PATH_INFO: `/v3/roles/service` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:40.023813 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2a25f25-f15a-4db2-811d-8b95bb10c5ce None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:39:40.035488 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-c2a25f25-f15a-4db2-811d-8b95bb10c5ce None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 11 00:39:40.035488 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:40.035488 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:40.035488 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:40.035488 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:40.035488 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:40.035488 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:40.035488 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:40.035488 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:40.035488 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:40.035488 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:40.035488 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:40.035488 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:39:40.035488 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:39:40.035488 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:39:40.035488 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:39:40.036688 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:39:40.036688 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:39:40.036688 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:39:40.036688 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:39:40.036688 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:40.036688 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:40.036688 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:40.036688 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:40.036688 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:40.036688 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:40.036688 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:40.036688 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:40.036688 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:40.036688 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:40.036688 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:40.036688 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:40.037512 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:40.037512 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:40.037512 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:40.037512 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:40.037512 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:39:40.037512 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:39:40.037512 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:39:40.037512 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:39:40.037512 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:39:40.037512 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:39:40.037512 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 11 00:39:40.037512 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:40.047214 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 283/552] 10.222.0.27 () {56 vars in 1102 bytes} [Sun Sep 11 00:39:39 2022] GET /identity/v3/roles/service => generated 85 bytes in 301 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:39:40.050493 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a25f25-f15a-4db2-811d-8b95bb10c5ce None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:40.120013 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5bae3d-4da3-49b4-ac45-3626ead15c73 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:40.136155 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:40.172919 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519156ca-2efa-43c2-b126-234dd8877e4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:40.173887 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519156ca-2efa-43c2-b126-234dd8877e4b None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:40.174342 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519156ca-2efa-43c2-b126-234dd8877e4b None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:40.174911 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519156ca-2efa-43c2-b126-234dd8877e4b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:40.210361 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:40.214191 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a25f25-f15a-4db2-811d-8b95bb10c5ce None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:40.250737 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19637496-a58c-4ce1-a2e8-644263cb6138 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:40.251375 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19637496-a58c-4ce1-a2e8-644263cb6138 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:40.251652 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19637496-a58c-4ce1-a2e8-644263cb6138 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:40.251926 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19637496-a58c-4ce1-a2e8-644263cb6138 None admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:40.252588 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19637496-a58c-4ce1-a2e8-644263cb6138 None admin] RBAC: Adding query filter params (name=service) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:40.252888 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19637496-a58c-4ce1-a2e8-644263cb6138 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:40.254332 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19637496-a58c-4ce1-a2e8-644263cb6138 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:40.258051 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519156ca-2efa-43c2-b126-234dd8877e4b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:40.259544 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519156ca-2efa-43c2-b126-234dd8877e4b None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:40.262849 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 284/553] 10.222.0.27 () {56 vars in 1103 bytes} [Sun Sep 11 00:39:40 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:39:40.264043 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055ac9d7-7771-4507-9682-323593de5867 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:40.264326 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055ac9d7-7771-4507-9682-323593de5867 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:40.264597 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055ac9d7-7771-4507-9682-323593de5867 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:40.265947 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 285/554] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:39:40 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:39:40.266803 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25af024f-eeb1-48c3-9f9a-16ce4c8451bf None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:40.267126 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25af024f-eeb1-48c3-9f9a-16ce4c8451bf None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:40.267403 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25af024f-eeb1-48c3-9f9a-16ce4c8451bf None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:40.268671 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 286/555] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:39:40 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:39:40.271611 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25af024f-eeb1-48c3-9f9a-16ce4c8451bf None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:40.373277 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 270/556] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:39:39 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 462 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 11 00:39:40.374532 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598b8333-f63e-41e7-84f8-db076a0409c7 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:40.374838 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598b8333-f63e-41e7-84f8-db076a0409c7 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:40.375113 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598b8333-f63e-41e7-84f8-db076a0409c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:40.381728 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:40.389354 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25af024f-eeb1-48c3-9f9a-16ce4c8451bf None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:40.395699 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-598b8333-f63e-41e7-84f8-db076a0409c7 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:39:40.396894 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-598b8333-f63e-41e7-84f8-db076a0409c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:39:40.424043 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a26f637-abd7-4d86-bb4c-7b8c8126202a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:40.424668 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a26f637-abd7-4d86-bb4c-7b8c8126202a None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:40.424944 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a26f637-abd7-4d86-bb4c-7b8c8126202a None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:40.425208 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a26f637-abd7-4d86-bb4c-7b8c8126202a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:40.437241 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a26f637-abd7-4d86-bb4c-7b8c8126202a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:40.438692 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a26f637-abd7-4d86-bb4c-7b8c8126202a None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:40.453852 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-2a26f637-abd7-4d86-bb4c-7b8c8126202a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:40.453852 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:40.453852 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:40.453852 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:40.453852 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:40.453852 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:40.453852 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:40.453852 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:40.453852 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:40.453852 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:40.453852 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:40.453852 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:40.453852 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:39:40.453852 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:39:40.453852 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:39:40.453852 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:39:40.454928 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:40.454928 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:40.454928 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:40.454928 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:40.454928 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:40.454928 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:40.454928 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:40.454928 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:40.454928 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:40.454928 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:40.454928 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:40.454928 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:40.454928 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:40.454928 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:40.454928 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:40.454928 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:40.454928 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:39:40.461132 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:39:40.461132 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:40.461132 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:40.461132 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 287/557] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:39:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:40.461132 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e136d6a6-934a-440c-a1f7-8bf27059779c None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:40.461132 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e136d6a6-934a-440c-a1f7-8bf27059779c None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:40.461132 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e136d6a6-934a-440c-a1f7-8bf27059779c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:40.464191 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:40.475438 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-e136d6a6-934a-440c-a1f7-8bf27059779c None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:39:40.476728 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-e136d6a6-934a-440c-a1f7-8bf27059779c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:39:40.477156 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 271/558] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:39:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:39:40.478212 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598b8333-f63e-41e7-84f8-db076a0409c7 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:40.560720 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 288/559] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:39:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:39:40.561702 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e136d6a6-934a-440c-a1f7-8bf27059779c None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:40.599205 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598b8333-f63e-41e7-84f8-db076a0409c7 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:40.652392 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:40.676330 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b9134a3-9005-40fe-bfdc-4b0607476bd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:40.676989 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b9134a3-9005-40fe-bfdc-4b0607476bd2 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:40.677267 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b9134a3-9005-40fe-bfdc-4b0607476bd2 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:40.677597 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b9134a3-9005-40fe-bfdc-4b0607476bd2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:40.677870 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e136d6a6-934a-440c-a1f7-8bf27059779c None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:40.699858 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b9134a3-9005-40fe-bfdc-4b0607476bd2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:40.701329 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b9134a3-9005-40fe-bfdc-4b0607476bd2 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:40.717626 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6131b51a-559e-461f-b254-34efd4e01c2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:40.718546 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6131b51a-559e-461f-b254-34efd4e01c2e None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:40.719121 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6131b51a-559e-461f-b254-34efd4e01c2e None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:40.720040 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6131b51a-559e-461f-b254-34efd4e01c2e None admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:40.720894 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6131b51a-559e-461f-b254-34efd4e01c2e None admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:40.721408 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6131b51a-559e-461f-b254-34efd4e01c2e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:40.723422 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6131b51a-559e-461f-b254-34efd4e01c2e None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:40.728020 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-2b9134a3-9005-40fe-bfdc-4b0607476bd2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:40.728020 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:40.728020 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:40.728020 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:40.728020 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:40.728020 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:40.728020 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:40.728020 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:40.728020 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:40.728020 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:40.728020 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:40.728020 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:40.728020 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:39:40.728020 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:39:40.728020 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:39:40.728020 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:39:40.728957 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:40.728957 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:40.728957 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:40.728957 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:40.728957 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:40.728957 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:40.728957 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:40.728957 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:40.728957 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:40.728957 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:40.728957 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:40.728957 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:40.728957 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:40.728957 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:40.728957 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:40.728957 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:40.728957 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:39:40.730259 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:39:40.730259 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:40.730259 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:40.730259 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 272/560] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:39:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 252 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:40.734976 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b9134a3-9005-40fe-bfdc-4b0607476bd2 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:40.744411 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 289/561] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:39:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:40.746009 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6131b51a-559e-461f-b254-34efd4e01c2e None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:40.821449 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:40.854873 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6131b51a-559e-461f-b254-34efd4e01c2e None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:40.871250 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b9134a3-9005-40fe-bfdc-4b0607476bd2 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:40.889101 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c5c2203-ad74-4387-ac60-a60e6d6bc79a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:40.889726 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5c2203-ad74-4387-ac60-a60e6d6bc79a None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:40.889999 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5c2203-ad74-4387-ac60-a60e6d6bc79a None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:40.890268 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5c2203-ad74-4387-ac60-a60e6d6bc79a None admin] PATH_INFO: `/v3/users/nova` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:40.897498 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c5c2203-ad74-4387-ac60-a60e6d6bc79a None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:40.899399 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c5c2203-ad74-4387-ac60-a60e6d6bc79a None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:40.902351 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:40.906160 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0860ab7-1260-469d-9d53-2cd08ab40dfa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:40.906754 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0860ab7-1260-469d-9d53-2cd08ab40dfa None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:40.907057 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0860ab7-1260-469d-9d53-2cd08ab40dfa None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:40.907323 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0860ab7-1260-469d-9d53-2cd08ab40dfa None admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:40.908069 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0860ab7-1260-469d-9d53-2cd08ab40dfa None admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:40.908423 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0860ab7-1260-469d-9d53-2cd08ab40dfa None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:40.909812 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0860ab7-1260-469d-9d53-2cd08ab40dfa None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:40.912670 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-9c5c2203-ad74-4387-ac60-a60e6d6bc79a None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 11 00:39:40.912670 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:40.912670 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:40.912670 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:40.912670 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:40.912670 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:40.912670 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:40.912670 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:40.912670 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:40.912670 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:40.912670 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:40.912670 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:40.912670 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 11 00:39:40.912670 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 11 00:39:40.912670 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 11 00:39:40.912670 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 11 00:39:40.914383 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:40.914383 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:40.914383 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 11 00:39:40.914383 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:39:40.914383 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 11 00:39:40.914383 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:39:40.914383 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:40.914383 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:40.914383 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:40.914383 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:40.914383 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:40.914383 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:40.914383 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:40.914383 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:40.914383 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:40.914383 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:40.914383 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:40.914383 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:40.915633 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:40.915633 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:40.915633 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 11 00:39:40.915633 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 11 00:39:40.915633 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 11 00:39:40.915633 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 11 00:39:40.915633 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 11 00:39:40.915633 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 11 00:39:40.915633 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 11 00:39:40.915633 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:40.915633 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 290/562] 10.222.0.27 () {56 vars in 1093 bytes} [Sun Sep 11 00:39:40 2022] GET /identity/v3/users/nova => generated 82 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:40.916224 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4678e5a8-b358-484d-93d5-974ce0993fb2 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:40.916224 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4678e5a8-b358-484d-93d5-974ce0993fb2 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:40.916609 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4678e5a8-b358-484d-93d5-974ce0993fb2 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:40.917877 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 291/563] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:39:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:39:40.918363 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4678e5a8-b358-484d-93d5-974ce0993fb2 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:40.934318 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 273/564] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:39:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:40.934318 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb09865-2135-4bc3-a8e1-8b81ac8a49bf None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:40.934318 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb09865-2135-4bc3-a8e1-8b81ac8a49bf None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:40.934318 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb09865-2135-4bc3-a8e1-8b81ac8a49bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:40.949946 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-7eb09865-2135-4bc3-a8e1-8b81ac8a49bf None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:39:40.951150 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-7eb09865-2135-4bc3-a8e1-8b81ac8a49bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:39:41.018747 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 274/565] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:39:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:39:41.019463 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb09865-2135-4bc3-a8e1-8b81ac8a49bf None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:41.039373 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4678e5a8-b358-484d-93d5-974ce0993fb2 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:41.052432 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:41.082143 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6293482a-e285-42fe-a897-0bfa0851e508 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:41.082928 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6293482a-e285-42fe-a897-0bfa0851e508 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:41.083207 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6293482a-e285-42fe-a897-0bfa0851e508 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:41.083500 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6293482a-e285-42fe-a897-0bfa0851e508 None admin] PATH_INFO: `/v3/roles/service` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:41.095116 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6293482a-e285-42fe-a897-0bfa0851e508 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:39:41.102579 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-6293482a-e285-42fe-a897-0bfa0851e508 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 11 00:39:41.102579 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:41.102579 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:41.102579 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:41.102579 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:41.102579 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:41.102579 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:41.102579 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:41.102579 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:41.102579 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:41.102579 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:41.102579 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:41.102579 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:39:41.102579 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:39:41.102579 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:39:41.102579 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:39:41.103598 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:39:41.103598 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:39:41.103598 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:39:41.103598 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:39:41.103598 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:41.103598 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:41.103598 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:41.103598 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:41.103598 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:41.103598 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:41.103598 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:41.103598 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:41.103598 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:41.103598 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:41.103598 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:41.103598 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:41.104511 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:41.104511 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:41.104511 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:41.104511 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:41.104511 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:39:41.104511 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:39:41.104511 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:39:41.104511 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:39:41.104511 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:39:41.104511 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:39:41.104511 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 11 00:39:41.104511 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:41.107032 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 292/566] 10.222.0.27 () {56 vars in 1102 bytes} [Sun Sep 11 00:39:40 2022] GET /identity/v3/roles/service => generated 85 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:41.108430 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6293482a-e285-42fe-a897-0bfa0851e508 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:41.126076 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb09865-2135-4bc3-a8e1-8b81ac8a49bf None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:41.163296 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff8b5fb4-3fe3-40c8-a311-442684106580 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:41.164279 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8b5fb4-3fe3-40c8-a311-442684106580 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:41.164565 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8b5fb4-3fe3-40c8-a311-442684106580 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:41.164836 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8b5fb4-3fe3-40c8-a311-442684106580 None admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:41.165743 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff8b5fb4-3fe3-40c8-a311-442684106580 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:41.166347 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff8b5fb4-3fe3-40c8-a311-442684106580 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:41.167854 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff8b5fb4-3fe3-40c8-a311-442684106580 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:41.174890 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:41.188568 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 275/567] 10.222.0.27 () {56 vars in 1133 bytes} [Sun Sep 11 00:39:41 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:41.191691 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff8b5fb4-3fe3-40c8-a311-442684106580 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:41.221174 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6293482a-e285-42fe-a897-0bfa0851e508 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:41.256335 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf505f0-76f2-464a-a69c-784b60807244 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:41.256965 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf505f0-76f2-464a-a69c-784b60807244 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:41.257267 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf505f0-76f2-464a-a69c-784b60807244 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:41.257748 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf505f0-76f2-464a-a69c-784b60807244 None admin] PATH_INFO: `/v3/projects/service` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:41.264709 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bf505f0-76f2-464a-a69c-784b60807244 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:41.266154 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bf505f0-76f2-464a-a69c-784b60807244 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:41.278407 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-6bf505f0-76f2-464a-a69c-784b60807244 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:39:41.278407 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:41.278407 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:41.278407 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:41.278407 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:41.278407 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:41.278407 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:41.278407 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:41.278407 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:41.278407 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:41.278407 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:41.278407 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:41.278407 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 11 00:39:41.278407 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 11 00:39:41.278407 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 11 00:39:41.279864 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 11 00:39:41.279864 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:41.279864 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:41.279864 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:41.279864 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:41.279864 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:41.279864 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:41.279864 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:41.279864 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:41.279864 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:41.279864 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:41.279864 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:41.279864 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:41.279864 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:41.279864 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:41.279864 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:41.279864 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:41.280745 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 11 00:39:41.280745 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 11 00:39:41.280745 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 11 00:39:41.280745 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 11 00:39:41.280745 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 11 00:39:41.280745 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 11 00:39:41.280745 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:39:41.280745 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:41.280745 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 293/568] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:39:41 2022] GET /identity/v3/projects/service => generated 88 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:41.280745 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf505f0-76f2-464a-a69c-784b60807244 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:41.297065 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:41.306869 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff8b5fb4-3fe3-40c8-a311-442684106580 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:41.343256 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf8115ce-9a7f-4161-8ce8-d7d4b00c449a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:41.343892 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf8115ce-9a7f-4161-8ce8-d7d4b00c449a None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:41.344163 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf8115ce-9a7f-4161-8ce8-d7d4b00c449a None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:41.344423 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf8115ce-9a7f-4161-8ce8-d7d4b00c449a None admin] PATH_INFO: `/v3/roles/service` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:41.355955 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf8115ce-9a7f-4161-8ce8-d7d4b00c449a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:39:41.360821 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-bf8115ce-9a7f-4161-8ce8-d7d4b00c449a None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 11 00:39:41.360821 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:41.360821 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:41.360821 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:41.360821 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:41.360821 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:41.360821 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:41.360821 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:41.360821 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:41.360821 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:41.360821 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:41.360821 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:41.360821 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:39:41.360821 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:39:41.360821 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:39:41.360821 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:39:41.361848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:39:41.361848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:39:41.361848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:39:41.361848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:39:41.361848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:41.361848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:41.361848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:41.361848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:41.361848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:41.361848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:41.361848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:41.361848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:41.361848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:41.361848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:41.361848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:41.361848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:41.362679 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:41.362679 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:41.362679 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:41.362679 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:41.362679 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:39:41.362679 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:39:41.362679 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:39:41.362679 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:39:41.362679 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:39:41.362679 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:39:41.362679 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 11 00:39:41.362679 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:41.362679 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 276/569] 10.222.0.27 () {56 vars in 1102 bytes} [Sun Sep 11 00:39:41 2022] GET /identity/v3/roles/service => generated 85 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:41.362679 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf8115ce-9a7f-4161-8ce8-d7d4b00c449a None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:41.388106 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf505f0-76f2-464a-a69c-784b60807244 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:41.407528 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:41.422938 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4884d6c-6db8-4d38-a328-3c0703516f79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:41.423479 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4884d6c-6db8-4d38-a328-3c0703516f79 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:41.423943 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4884d6c-6db8-4d38-a328-3c0703516f79 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:41.424215 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4884d6c-6db8-4d38-a328-3c0703516f79 None admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:41.425187 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4884d6c-6db8-4d38-a328-3c0703516f79 None admin] RBAC: Adding query filter params (name=service) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:41.425490 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4884d6c-6db8-4d38-a328-3c0703516f79 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:41.426885 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4884d6c-6db8-4d38-a328-3c0703516f79 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:41.435744 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 294/570] 10.222.0.27 () {56 vars in 1103 bytes} [Sun Sep 11 00:39:41 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:41.436036 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4884d6c-6db8-4d38-a328-3c0703516f79 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:41.465452 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf8115ce-9a7f-4161-8ce8-d7d4b00c449a None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:41.499507 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6125e1a-7603-4113-a9d8-5827154355d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:41.500124 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6125e1a-7603-4113-a9d8-5827154355d4 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:41.500391 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6125e1a-7603-4113-a9d8-5827154355d4 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:41.500661 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6125e1a-7603-4113-a9d8-5827154355d4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:41.510811 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6125e1a-7603-4113-a9d8-5827154355d4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:41.512267 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6125e1a-7603-4113-a9d8-5827154355d4 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:41.526634 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-a6125e1a-7603-4113-a9d8-5827154355d4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:41.526634 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:41.526634 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:41.526634 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:41.526634 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:41.526634 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:41.526634 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:41.526634 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:41.526634 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:41.526634 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:41.526634 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:41.526634 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:41.526634 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:39:41.526634 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:39:41.526634 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:39:41.526634 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:39:41.527819 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:41.527819 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:41.527819 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:41.527819 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:41.527819 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:41.527819 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:41.527819 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:41.527819 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:41.527819 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:41.527819 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:41.527819 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:41.527819 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:41.527819 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:41.527819 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:41.527819 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:41.527819 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:41.527819 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:39:41.528704 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:39:41.528704 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:41.528704 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:41.528704 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 277/571] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:39:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:41.528704 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6125e1a-7603-4113-a9d8-5827154355d4 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:41.531199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:41.539222 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4884d6c-6db8-4d38-a328-3c0703516f79 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:41.573810 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c30ec05b-e8b9-4ada-8d12-0f63b2ac64cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:41.574561 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c30ec05b-e8b9-4ada-8d12-0f63b2ac64cb None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:41.574863 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c30ec05b-e8b9-4ada-8d12-0f63b2ac64cb None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:41.575136 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c30ec05b-e8b9-4ada-8d12-0f63b2ac64cb None admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:41.575822 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c30ec05b-e8b9-4ada-8d12-0f63b2ac64cb None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:41.576122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c30ec05b-e8b9-4ada-8d12-0f63b2ac64cb None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:41.577539 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c30ec05b-e8b9-4ada-8d12-0f63b2ac64cb None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:41.590196 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 295/572] 10.222.0.27 () {56 vars in 1148 bytes} [Sun Sep 11 00:39:41 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:41.591008 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c30ec05b-e8b9-4ada-8d12-0f63b2ac64cb None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:41.633858 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6125e1a-7603-4113-a9d8-5827154355d4 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:41.634038 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:41.668236 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e89df0be-2f31-48a0-9734-538db69b4023 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:41.668861 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89df0be-2f31-48a0-9734-538db69b4023 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:41.669134 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89df0be-2f31-48a0-9734-538db69b4023 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:41.669403 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89df0be-2f31-48a0-9734-538db69b4023 None admin] PATH_INFO: `/v3/roles/service` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:41.679908 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e89df0be-2f31-48a0-9734-538db69b4023 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:39:41.685770 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-e89df0be-2f31-48a0-9734-538db69b4023 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 11 00:39:41.685770 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:41.685770 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:41.685770 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:41.685770 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:41.685770 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:41.685770 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:41.685770 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:41.685770 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:41.685770 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:41.685770 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:41.685770 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:41.685770 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:39:41.685770 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:39:41.685770 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:39:41.685770 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:39:41.686701 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:39:41.686701 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:39:41.686701 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:39:41.686701 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:39:41.686701 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:41.686701 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:41.686701 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:41.686701 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:41.686701 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:41.686701 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:41.686701 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:41.686701 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:41.686701 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:41.686701 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:41.686701 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:41.686701 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:41.687617 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:41.687617 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:41.687617 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:41.687617 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:41.687617 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:39:41.687617 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:39:41.687617 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:39:41.687617 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:39:41.687617 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:39:41.687617 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:39:41.687617 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 11 00:39:41.687617 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:41.687617 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 278/573] 10.222.0.27 () {56 vars in 1102 bytes} [Sun Sep 11 00:39:41 2022] GET /identity/v3/roles/service => generated 85 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:41.688324 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e89df0be-2f31-48a0-9734-538db69b4023 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:41.692316 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c30ec05b-e8b9-4ada-8d12-0f63b2ac64cb None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:41.727001 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c1f747d-f818-478f-9eeb-358654f74c9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:41.727600 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1f747d-f818-478f-9eeb-358654f74c9d None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:41.727875 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1f747d-f818-478f-9eeb-358654f74c9d None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:41.728144 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1f747d-f818-478f-9eeb-358654f74c9d None admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:41.728800 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c1f747d-f818-478f-9eeb-358654f74c9d None admin] RBAC: Adding query filter params (name=service) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:41.729100 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c1f747d-f818-478f-9eeb-358654f74c9d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:41.730455 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c1f747d-f818-478f-9eeb-358654f74c9d None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:41.738193 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 296/574] 10.222.0.27 () {56 vars in 1103 bytes} [Sun Sep 11 00:39:41 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:41.738982 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c1f747d-f818-478f-9eeb-358654f74c9d None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:41.748829 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:41.787974 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e89df0be-2f31-48a0-9734-538db69b4023 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:41.822396 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58661042-ed6d-41fe-8e99-1a1ba9c11a22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:41.823057 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58661042-ed6d-41fe-8e99-1a1ba9c11a22 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:41.823333 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58661042-ed6d-41fe-8e99-1a1ba9c11a22 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:41.823602 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58661042-ed6d-41fe-8e99-1a1ba9c11a22 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:41.833840 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58661042-ed6d-41fe-8e99-1a1ba9c11a22 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:41.835345 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58661042-ed6d-41fe-8e99-1a1ba9c11a22 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:41.838990 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c1f747d-f818-478f-9eeb-358654f74c9d None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:41.841255 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:41.850318 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-58661042-ed6d-41fe-8e99-1a1ba9c11a22 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:41.850318 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:41.850318 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:41.850318 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:41.850318 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:41.850318 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:41.850318 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:41.850318 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:41.850318 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:41.850318 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:41.850318 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:41.850318 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:41.850318 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:39:41.850318 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:39:41.850318 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:39:41.850318 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:39:41.851642 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:41.851642 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:41.851642 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:41.851642 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:41.851642 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:41.851642 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:41.851642 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:41.851642 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:41.851642 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:41.851642 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:41.851642 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:41.851642 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:41.851642 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:41.851642 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:41.851642 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:41.851642 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:41.851642 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:39:41.852552 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:39:41.852552 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:41.852552 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:41.852552 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 279/575] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:39:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:41.852552 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58661042-ed6d-41fe-8e99-1a1ba9c11a22 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:41.877534 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f647eb1-c34d-468a-9aa0-6430072dcf31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:41.878131 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f647eb1-c34d-468a-9aa0-6430072dcf31 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:41.878404 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f647eb1-c34d-468a-9aa0-6430072dcf31 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:41.878675 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f647eb1-c34d-468a-9aa0-6430072dcf31 None admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:41.879337 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f647eb1-c34d-468a-9aa0-6430072dcf31 None admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:41.879633 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f647eb1-c34d-468a-9aa0-6430072dcf31 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:41.881000 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f647eb1-c34d-468a-9aa0-6430072dcf31 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:41.892567 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 297/576] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:39:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:41.893336 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f647eb1-c34d-468a-9aa0-6430072dcf31 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:41.955577 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58661042-ed6d-41fe-8e99-1a1ba9c11a22 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:41.969276 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:41.989523 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c153141-8113-41b4-8b6e-fbe020da9d41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:41.990119 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c153141-8113-41b4-8b6e-fbe020da9d41 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:41.990390 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c153141-8113-41b4-8b6e-fbe020da9d41 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:41.990659 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c153141-8113-41b4-8b6e-fbe020da9d41 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:41.991400 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c153141-8113-41b4-8b6e-fbe020da9d41 None admin] RBAC: Adding query filter params (role.id=0390f926e7ef4715ae18697033bde90e, scope.project.id=b504a5058bc6474c8cc077dfeec16799, user.id=bcb73dec73464a3abc4b774abb4f416f) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:41.991803 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c153141-8113-41b4-8b6e-fbe020da9d41 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:41.992228 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f647eb1-c34d-468a-9aa0-6430072dcf31 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:41.993233 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c153141-8113-41b4-8b6e-fbe020da9d41 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:41.998634 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 280/577] 10.222.0.27 () {56 vars in 1374 bytes} [Sun Sep 11 00:39:41 2022] GET /identity/v3/role_assignments?user.id=bcb73dec73464a3abc4b774abb4f416f&scope.project.id=b504a5058bc6474c8cc077dfeec16799&role.id=0390f926e7ef4715ae18697033bde90e => generated 259 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:41.999432 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c153141-8113-41b4-8b6e-fbe020da9d41 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:42.041049 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44b71661-4e34-413a-815e-edd0c674d4a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:42.041699 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b71661-4e34-413a-815e-edd0c674d4a0 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:42.041975 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b71661-4e34-413a-815e-edd0c674d4a0 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:42.042242 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b71661-4e34-413a-815e-edd0c674d4a0 None admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:42.042946 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44b71661-4e34-413a-815e-edd0c674d4a0 None admin] RBAC: Adding query filter params (name=service) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:42.043246 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44b71661-4e34-413a-815e-edd0c674d4a0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:42.044632 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44b71661-4e34-413a-815e-edd0c674d4a0 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:42.058454 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 298/578] 10.222.0.27 () {56 vars in 1103 bytes} [Sun Sep 11 00:39:41 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:42.058454 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44b71661-4e34-413a-815e-edd0c674d4a0 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:42.092947 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:42.131338 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c153141-8113-41b4-8b6e-fbe020da9d41 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:42.170258 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3242e6e-59c9-4552-b183-567e39117101 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:42.170925 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3242e6e-59c9-4552-b183-567e39117101 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:42.171200 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3242e6e-59c9-4552-b183-567e39117101 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:42.171469 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3242e6e-59c9-4552-b183-567e39117101 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:42.182504 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44b71661-4e34-413a-815e-edd0c674d4a0 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:42.194852 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3242e6e-59c9-4552-b183-567e39117101 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:42.194852 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3242e6e-59c9-4552-b183-567e39117101 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:42.201093 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-c3242e6e-59c9-4552-b183-567e39117101 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:42.201093 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:42.201093 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:42.201093 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:42.201093 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:42.201093 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:42.201093 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:42.201093 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:42.201093 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:42.201093 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:42.201093 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:42.201093 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:42.201093 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:39:42.201093 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:39:42.201093 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:39:42.201093 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:39:42.202199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:42.202199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:42.202199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:42.202199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:42.202199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:42.202199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:42.202199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:42.202199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:42.202199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:42.202199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:42.202199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:42.202199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:42.202199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:42.202199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:42.202199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:42.202199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:42.202199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:39:42.203322 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:39:42.203322 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:42.203322 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:42.206889 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 281/579] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:39:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:42.207670 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3242e6e-59c9-4552-b183-567e39117101 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:42.216722 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-005a3997-c095-4c7c-8987-cda381973168 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:42.217322 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005a3997-c095-4c7c-8987-cda381973168 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:42.217599 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005a3997-c095-4c7c-8987-cda381973168 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:42.217862 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005a3997-c095-4c7c-8987-cda381973168 None admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:42.218501 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-005a3997-c095-4c7c-8987-cda381973168 None admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:42.218815 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-005a3997-c095-4c7c-8987-cda381973168 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:42.220177 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-005a3997-c095-4c7c-8987-cda381973168 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:42.233044 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 299/580] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:39:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:42.233847 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-005a3997-c095-4c7c-8987-cda381973168 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:42.244697 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:42.340719 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3242e6e-59c9-4552-b183-567e39117101 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:42.352725 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:42.377161 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-005a3997-c095-4c7c-8987-cda381973168 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:42.392290 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c80fbf-2877-4aaa-80b1-b3e459439180 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:42.393247 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c80fbf-2877-4aaa-80b1-b3e459439180 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:42.393701 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c80fbf-2877-4aaa-80b1-b3e459439180 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:42.394148 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c80fbf-2877-4aaa-80b1-b3e459439180 None admin] PATH_INFO: `/v3/projects/service` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:42.402565 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7c80fbf-2877-4aaa-80b1-b3e459439180 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:42.404060 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7c80fbf-2877-4aaa-80b1-b3e459439180 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:42.416531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-e7c80fbf-2877-4aaa-80b1-b3e459439180 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:39:42.416531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:42.416531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:42.416531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:42.416531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:42.416531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:42.416531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:42.416531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:42.416531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:42.416531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:42.416531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:42.416531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:42.416531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 11 00:39:42.416531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 11 00:39:42.416531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 11 00:39:42.417454 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 11 00:39:42.417454 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:42.417454 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:42.417454 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:42.417454 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:42.417454 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:42.417454 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:42.417454 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:42.417454 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:42.417454 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:42.417454 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:42.417454 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:42.417454 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:42.417454 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:42.417454 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:42.417454 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:42.417454 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:42.418472 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 11 00:39:42.418472 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 11 00:39:42.418472 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 11 00:39:42.418472 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 11 00:39:42.418472 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 11 00:39:42.418472 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 11 00:39:42.418472 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:39:42.418472 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:42.422238 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 282/581] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:39:42 2022] GET /identity/v3/projects/service => generated 88 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:42.422958 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c80fbf-2877-4aaa-80b1-b3e459439180 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:42.423557 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e072bc88-0c28-4fac-94dc-31dc33a0700f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:42.424184 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e072bc88-0c28-4fac-94dc-31dc33a0700f None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:42.424458 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e072bc88-0c28-4fac-94dc-31dc33a0700f None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:42.424729 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e072bc88-0c28-4fac-94dc-31dc33a0700f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:42.436994 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e072bc88-0c28-4fac-94dc-31dc33a0700f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:42.438430 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e072bc88-0c28-4fac-94dc-31dc33a0700f None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:42.453859 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-e072bc88-0c28-4fac-94dc-31dc33a0700f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:42.453859 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:42.453859 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:42.453859 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:42.453859 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:42.453859 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:42.453859 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:42.453859 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:42.453859 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:42.453859 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:42.453859 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:42.453859 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:42.453859 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:39:42.453859 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:39:42.453859 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:39:42.453859 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:39:42.454952 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:42.454952 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:42.454952 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:42.454952 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:42.454952 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:42.454952 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:42.454952 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:42.454952 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:42.454952 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:42.454952 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:42.454952 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:42.454952 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:42.454952 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:42.454952 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:42.454952 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:42.454952 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:42.454952 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:39:42.456081 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:39:42.456081 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:42.456081 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:42.459636 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 300/582] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:39:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:42.460488 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e072bc88-0c28-4fac-94dc-31dc33a0700f None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:42.521682 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:42.533722 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c80fbf-2877-4aaa-80b1-b3e459439180 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:42.569737 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b079c449-0663-460a-8d7f-ceffc28b2a08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:42.570356 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b079c449-0663-460a-8d7f-ceffc28b2a08 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:42.570686 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b079c449-0663-460a-8d7f-ceffc28b2a08 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:42.570984 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b079c449-0663-460a-8d7f-ceffc28b2a08 None admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:42.571493 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e072bc88-0c28-4fac-94dc-31dc33a0700f None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:42.571663 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b079c449-0663-460a-8d7f-ceffc28b2a08 None admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:42.571926 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b079c449-0663-460a-8d7f-ceffc28b2a08 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:42.573294 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b079c449-0663-460a-8d7f-ceffc28b2a08 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:42.590332 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 283/583] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:39:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:42.590332 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b079c449-0663-460a-8d7f-ceffc28b2a08 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:42.609036 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b778e0f-322d-412e-b011-8a13706cfa82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:42.609665 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b778e0f-322d-412e-b011-8a13706cfa82 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:42.609942 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b778e0f-322d-412e-b011-8a13706cfa82 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:42.610209 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b778e0f-322d-412e-b011-8a13706cfa82 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:42.618490 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b778e0f-322d-412e-b011-8a13706cfa82 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:42.620024 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b778e0f-322d-412e-b011-8a13706cfa82 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:42.621805 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:42.633815 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-6b778e0f-322d-412e-b011-8a13706cfa82 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 11 00:39:42.633815 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:42.633815 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:42.633815 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:42.633815 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:42.633815 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:42.633815 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:42.633815 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:42.633815 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:42.633815 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:42.633815 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:42.633815 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:42.633815 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 11 00:39:42.633815 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 11 00:39:42.633815 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 11 00:39:42.633815 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 11 00:39:42.634806 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:42.634806 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:42.634806 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 11 00:39:42.634806 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:39:42.634806 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 11 00:39:42.634806 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:39:42.634806 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:42.634806 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:42.634806 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:42.634806 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:42.634806 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:42.634806 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:42.634806 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:42.634806 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:42.634806 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:42.634806 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:42.634806 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:42.634806 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:42.636718 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:42.636718 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:42.636718 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 11 00:39:42.636718 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 11 00:39:42.636718 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 11 00:39:42.636718 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 11 00:39:42.636718 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 11 00:39:42.636718 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 11 00:39:42.636718 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 11 00:39:42.636718 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:42.638922 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 301/584] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:39:42 2022] GET /identity/v3/users/cinder => generated 84 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:42.640499 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b778e0f-322d-412e-b011-8a13706cfa82 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:42.699788 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b079c449-0663-460a-8d7f-ceffc28b2a08 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:42.735796 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8accfd55-02de-415c-9024-6be59c8c7af4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:42.736592 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8accfd55-02de-415c-9024-6be59c8c7af4 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:42.736874 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8accfd55-02de-415c-9024-6be59c8c7af4 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:42.737153 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8accfd55-02de-415c-9024-6be59c8c7af4 None admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:42.737999 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8accfd55-02de-415c-9024-6be59c8c7af4 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:42.738597 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8accfd55-02de-415c-9024-6be59c8c7af4 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:42.740133 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8accfd55-02de-415c-9024-6be59c8c7af4 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:42.758212 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 284/585] 10.222.0.27 () {56 vars in 1148 bytes} [Sun Sep 11 00:39:42 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:42.758212 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8accfd55-02de-415c-9024-6be59c8c7af4 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:42.758465 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b778e0f-322d-412e-b011-8a13706cfa82 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:42.770899 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:42.795081 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c00a175d-a4e4-4890-ba00-9f1e01831a2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:42.796280 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00a175d-a4e4-4890-ba00-9f1e01831a2b None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:42.796280 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00a175d-a4e4-4890-ba00-9f1e01831a2b None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:42.796280 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00a175d-a4e4-4890-ba00-9f1e01831a2b None admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:42.796853 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c00a175d-a4e4-4890-ba00-9f1e01831a2b None admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:42.797160 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c00a175d-a4e4-4890-ba00-9f1e01831a2b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:42.798542 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c00a175d-a4e4-4890-ba00-9f1e01831a2b None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:42.811214 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 302/586] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:39:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:42.812038 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c00a175d-a4e4-4890-ba00-9f1e01831a2b None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:42.874701 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:42.877064 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8accfd55-02de-415c-9024-6be59c8c7af4 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:42.913372 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6156a5aa-c5e8-458a-8154-d59a88dc58d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:42.914521 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6156a5aa-c5e8-458a-8154-d59a88dc58d6 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:42.914873 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6156a5aa-c5e8-458a-8154-d59a88dc58d6 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:42.915206 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6156a5aa-c5e8-458a-8154-d59a88dc58d6 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:42.923368 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6156a5aa-c5e8-458a-8154-d59a88dc58d6 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:42.924816 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6156a5aa-c5e8-458a-8154-d59a88dc58d6 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:42.925810 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c00a175d-a4e4-4890-ba00-9f1e01831a2b None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:42.936685 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-6156a5aa-c5e8-458a-8154-d59a88dc58d6 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 11 00:39:42.936685 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:42.936685 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:42.936685 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:42.936685 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:42.936685 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:42.936685 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:42.936685 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:42.936685 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:42.936685 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:42.936685 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:42.936685 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:42.936685 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 11 00:39:42.936685 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 11 00:39:42.936685 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 11 00:39:42.936685 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 11 00:39:42.937780 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:42.937780 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:42.937780 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 11 00:39:42.937780 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:39:42.937780 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 11 00:39:42.937780 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:39:42.937780 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:42.937780 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:42.937780 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:42.937780 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:42.937780 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:42.937780 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:42.937780 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:42.937780 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:42.937780 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:42.937780 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:42.937780 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:42.937780 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:42.938818 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:42.938818 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:42.938818 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 11 00:39:42.938818 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 11 00:39:42.938818 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 11 00:39:42.938818 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 11 00:39:42.938818 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 11 00:39:42.938818 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 11 00:39:42.938818 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 11 00:39:42.938818 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:42.938818 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 285/587] 10.222.0.27 () {56 vars in 1102 bytes} [Sun Sep 11 00:39:42 2022] GET /identity/v3/users/neutron => generated 85 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:42.938818 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6156a5aa-c5e8-458a-8154-d59a88dc58d6 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:42.969977 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc58e8c1-8ea2-4a2c-904b-7d5bd16c9a43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:42.970581 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc58e8c1-8ea2-4a2c-904b-7d5bd16c9a43 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:42.970885 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc58e8c1-8ea2-4a2c-904b-7d5bd16c9a43 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:42.971161 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc58e8c1-8ea2-4a2c-904b-7d5bd16c9a43 None admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:42.971884 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc58e8c1-8ea2-4a2c-904b-7d5bd16c9a43 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:42.972191 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc58e8c1-8ea2-4a2c-904b-7d5bd16c9a43 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:42.973616 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc58e8c1-8ea2-4a2c-904b-7d5bd16c9a43 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:42.997602 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 303/588] 10.222.0.27 () {56 vars in 1137 bytes} [Sun Sep 11 00:39:42 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:42.997602 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc58e8c1-8ea2-4a2c-904b-7d5bd16c9a43 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:43.015012 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:43.051447 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6156a5aa-c5e8-458a-8154-d59a88dc58d6 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:43.091987 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f25ae5ac-ae33-4858-91e3-6dccf0983c2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:43.092750 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25ae5ac-ae33-4858-91e3-6dccf0983c2e None admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:43.093032 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25ae5ac-ae33-4858-91e3-6dccf0983c2e None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:43.093309 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25ae5ac-ae33-4858-91e3-6dccf0983c2e None admin] PATH_INFO: `/v3/projects/b504a5058bc6474c8cc077dfeec16799/users/835def0e6c6d4365a2767051509eb0d6/roles/0390f926e7ef4715ae18697033bde90e` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:43.109896 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f25ae5ac-ae33-4858-91e3-6dccf0983c2e None admin] RBAC: Authorizing `identity:create_grant(project_id=b504a5058bc6474c8cc077dfeec16799, user_id=835def0e6c6d4365a2767051509eb0d6, role_id=0390f926e7ef4715ae18697033bde90e)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:43.111405 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f25ae5ac-ae33-4858-91e3-6dccf0983c2e None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:43.113639 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc58e8c1-8ea2-4a2c-904b-7d5bd16c9a43 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:43.124515 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 286/589] 10.222.0.27 () {58 vars in 1439 bytes} [Sun Sep 11 00:39:42 2022] PUT /identity/v3/projects/b504a5058bc6474c8cc077dfeec16799/users/835def0e6c6d4365a2767051509eb0d6/roles/0390f926e7ef4715ae18697033bde90e => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:39:43.125186 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f25ae5ac-ae33-4858-91e3-6dccf0983c2e None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:43.145376 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:43.169188 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d673311-fc53-4b8d-9df5-4fb1c1892251 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:43.170264 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d673311-fc53-4b8d-9df5-4fb1c1892251 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:43.170779 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d673311-fc53-4b8d-9df5-4fb1c1892251 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:43.171323 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d673311-fc53-4b8d-9df5-4fb1c1892251 None admin] PATH_INFO: `/v3/users/glance` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:43.181095 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d673311-fc53-4b8d-9df5-4fb1c1892251 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:43.183091 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d673311-fc53-4b8d-9df5-4fb1c1892251 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:43.197696 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-7d673311-fc53-4b8d-9df5-4fb1c1892251 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 11 00:39:43.197696 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:43.197696 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:43.197696 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:43.197696 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:43.197696 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:43.197696 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:43.197696 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:43.197696 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:43.197696 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:43.197696 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:43.197696 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:43.197696 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 11 00:39:43.197696 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 11 00:39:43.197696 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 11 00:39:43.197696 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 11 00:39:43.198807 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:43.198807 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:43.198807 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 11 00:39:43.198807 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:39:43.198807 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 11 00:39:43.198807 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:39:43.198807 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:43.198807 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:43.198807 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:43.198807 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:43.198807 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:43.198807 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:43.198807 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:43.198807 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:43.198807 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:43.198807 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:43.198807 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:43.198807 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:43.199774 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:43.199774 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:43.199774 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 11 00:39:43.199774 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 11 00:39:43.199774 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 11 00:39:43.199774 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 11 00:39:43.199774 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 11 00:39:43.199774 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 11 00:39:43.199774 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 11 00:39:43.199774 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:43.207237 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 304/590] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:39:42 2022] GET /identity/v3/users/glance => generated 84 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:43.208604 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d673311-fc53-4b8d-9df5-4fb1c1892251 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:43.261971 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f25ae5ac-ae33-4858-91e3-6dccf0983c2e None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:43.294083 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:43.314323 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd47ad20-4576-4e80-9db8-fff54166ebc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:43.315559 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd47ad20-4576-4e80-9db8-fff54166ebc0 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:43.316588 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd47ad20-4576-4e80-9db8-fff54166ebc0 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:43.317181 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd47ad20-4576-4e80-9db8-fff54166ebc0 None admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:43.318225 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd47ad20-4576-4e80-9db8-fff54166ebc0 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:43.319026 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd47ad20-4576-4e80-9db8-fff54166ebc0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:43.321108 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd47ad20-4576-4e80-9db8-fff54166ebc0 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:43.346125 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d673311-fc53-4b8d-9df5-4fb1c1892251 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:43.350241 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 287/591] 10.222.0.27 () {56 vars in 1139 bytes} [Sun Sep 11 00:39:43 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:43.350241 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd47ad20-4576-4e80-9db8-fff54166ebc0 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:43.390927 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6820dc23-1ef9-4aaf-b20d-aa1f364caf3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:43.391606 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6820dc23-1ef9-4aaf-b20d-aa1f364caf3c None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:43.392473 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6820dc23-1ef9-4aaf-b20d-aa1f364caf3c None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:43.392473 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6820dc23-1ef9-4aaf-b20d-aa1f364caf3c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:43.403706 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6820dc23-1ef9-4aaf-b20d-aa1f364caf3c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:43.406053 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6820dc23-1ef9-4aaf-b20d-aa1f364caf3c None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:43.422403 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-6820dc23-1ef9-4aaf-b20d-aa1f364caf3c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:43.422403 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:43.422403 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:43.422403 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:43.422403 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:43.422403 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:43.422403 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:43.422403 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:43.422403 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:43.422403 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:43.422403 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:43.422403 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:43.422403 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:39:43.422403 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:39:43.422403 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:39:43.422403 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:39:43.423436 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:43.423436 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:43.423436 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:43.423436 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:43.423436 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:43.423436 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:43.423436 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:43.423436 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:43.423436 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:43.423436 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:43.423436 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:43.423436 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:43.423436 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:43.423436 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:43.423436 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:43.423436 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:43.423436 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:39:43.424350 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:39:43.424350 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:43.424350 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:43.425046 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 305/592] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:39:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:43.425225 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6820dc23-1ef9-4aaf-b20d-aa1f364caf3c None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:43.462821 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:43.473786 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd47ad20-4576-4e80-9db8-fff54166ebc0 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:43.513522 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5d6849-924c-40c0-ac14-2c921c730618 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:43.514073 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5d6849-924c-40c0-ac14-2c921c730618 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:43.514399 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5d6849-924c-40c0-ac14-2c921c730618 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:43.514671 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5d6849-924c-40c0-ac14-2c921c730618 None admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:43.515480 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e5d6849-924c-40c0-ac14-2c921c730618 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:43.515843 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e5d6849-924c-40c0-ac14-2c921c730618 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:43.517277 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e5d6849-924c-40c0-ac14-2c921c730618 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:43.536697 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 288/593] 10.222.0.27 () {56 vars in 1137 bytes} [Sun Sep 11 00:39:43 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:43.537527 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5d6849-924c-40c0-ac14-2c921c730618 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:43.546414 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6820dc23-1ef9-4aaf-b20d-aa1f364caf3c None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:43.576058 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:43.596890 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6da5055-5bd6-4b7e-85c4-f499ab7f1803 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:43.598175 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6da5055-5bd6-4b7e-85c4-f499ab7f1803 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:43.598981 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6da5055-5bd6-4b7e-85c4-f499ab7f1803 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:43.599859 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6da5055-5bd6-4b7e-85c4-f499ab7f1803 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:43.614412 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6da5055-5bd6-4b7e-85c4-f499ab7f1803 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:43.616509 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6da5055-5bd6-4b7e-85c4-f499ab7f1803 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:43.638764 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-d6da5055-5bd6-4b7e-85c4-f499ab7f1803 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:43.638764 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:43.638764 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:43.638764 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:43.638764 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:43.638764 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:43.638764 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:43.638764 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:43.638764 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:43.638764 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:43.638764 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:43.638764 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:43.638764 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:39:43.638764 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:39:43.638764 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:39:43.638764 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:39:43.639791 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:43.639791 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:43.639791 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:43.639791 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:43.639791 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:43.639791 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:43.639791 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:43.639791 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:43.639791 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:43.639791 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:43.639791 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:43.639791 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:43.639791 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:43.639791 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:43.639791 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:43.639791 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:43.639791 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:39:43.640694 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:39:43.640694 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:43.640694 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:43.642934 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 306/594] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:39:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:43.643865 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6da5055-5bd6-4b7e-85c4-f499ab7f1803 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:43.654186 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5d6849-924c-40c0-ac14-2c921c730618 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:43.691658 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee68f0c2-e649-4fd7-9dac-14a4bbc2f064 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:43.692418 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee68f0c2-e649-4fd7-9dac-14a4bbc2f064 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:43.692795 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee68f0c2-e649-4fd7-9dac-14a4bbc2f064 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:43.693162 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee68f0c2-e649-4fd7-9dac-14a4bbc2f064 None admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:43.694013 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee68f0c2-e649-4fd7-9dac-14a4bbc2f064 None admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:43.694472 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee68f0c2-e649-4fd7-9dac-14a4bbc2f064 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:43.697030 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee68f0c2-e649-4fd7-9dac-14a4bbc2f064 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:43.719140 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 289/595] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:39:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:43.720189 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee68f0c2-e649-4fd7-9dac-14a4bbc2f064 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:43.747818 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:43.774563 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6da5055-5bd6-4b7e-85c4-f499ab7f1803 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:43.814102 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d25584f6-dd37-4b49-b692-da3c9927c91e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:43.815055 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25584f6-dd37-4b49-b692-da3c9927c91e None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:43.815496 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25584f6-dd37-4b49-b692-da3c9927c91e None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:43.815928 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25584f6-dd37-4b49-b692-da3c9927c91e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:43.829987 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d25584f6-dd37-4b49-b692-da3c9927c91e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:43.831718 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d25584f6-dd37-4b49-b692-da3c9927c91e None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:43.838357 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee68f0c2-e649-4fd7-9dac-14a4bbc2f064 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:43.847687 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-d25584f6-dd37-4b49-b692-da3c9927c91e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:43.847687 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:43.847687 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:43.847687 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:43.847687 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:43.847687 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:43.847687 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:43.847687 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:43.847687 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:43.847687 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:43.847687 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:43.847687 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:43.847687 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:39:43.847687 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:39:43.847687 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:39:43.847687 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:39:43.848695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:43.848695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:43.848695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:43.848695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:43.848695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:43.848695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:43.848695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:43.848695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:43.848695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:43.848695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:43.848695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:43.848695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:43.848695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:43.848695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:43.848695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:43.848695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:43.848695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:39:43.849584 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:39:43.849584 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:43.849584 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:43.849584 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 307/596] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:39:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:43.850273 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d25584f6-dd37-4b49-b692-da3c9927c91e None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:43.859174 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:43.886300 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ecc0bd8-f2a6-4123-a50a-f23c6f3e7d12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:43.886931 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ecc0bd8-f2a6-4123-a50a-f23c6f3e7d12 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:43.887208 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ecc0bd8-f2a6-4123-a50a-f23c6f3e7d12 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:43.887476 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ecc0bd8-f2a6-4123-a50a-f23c6f3e7d12 None admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:43.888119 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ecc0bd8-f2a6-4123-a50a-f23c6f3e7d12 None admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:43.888569 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ecc0bd8-f2a6-4123-a50a-f23c6f3e7d12 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:43.890219 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ecc0bd8-f2a6-4123-a50a-f23c6f3e7d12 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:43.910659 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 290/597] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:39:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:43.912469 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ecc0bd8-f2a6-4123-a50a-f23c6f3e7d12 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:43.983888 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d25584f6-dd37-4b49-b692-da3c9927c91e None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:44.018580 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:44.026622 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe9a8c3-215f-41e5-909b-66fb1ee25a24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:44.027233 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe9a8c3-215f-41e5-909b-66fb1ee25a24 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:44.027555 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe9a8c3-215f-41e5-909b-66fb1ee25a24 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:44.027828 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe9a8c3-215f-41e5-909b-66fb1ee25a24 None admin] PATH_INFO: `/v3/projects/service` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:44.036110 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbe9a8c3-215f-41e5-909b-66fb1ee25a24 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:44.037652 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbe9a8c3-215f-41e5-909b-66fb1ee25a24 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:44.049835 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ecc0bd8-f2a6-4123-a50a-f23c6f3e7d12 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:44.050778 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-fbe9a8c3-215f-41e5-909b-66fb1ee25a24 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:39:44.050778 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:44.050778 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:44.050778 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:44.050778 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:44.050778 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:44.050778 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:44.050778 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:44.050778 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:44.050778 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:44.050778 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:44.050778 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:44.050778 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 11 00:39:44.050778 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 11 00:39:44.050778 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 11 00:39:44.051902 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 11 00:39:44.051902 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:44.051902 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:44.051902 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:44.051902 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:44.051902 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:44.051902 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:44.051902 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:44.051902 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:44.051902 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:44.051902 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:44.051902 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:44.051902 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:44.051902 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:44.051902 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:44.051902 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:44.051902 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:44.052786 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 11 00:39:44.052786 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 11 00:39:44.052786 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 11 00:39:44.052786 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 11 00:39:44.052786 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 11 00:39:44.052786 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 11 00:39:44.052786 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:39:44.052786 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:44.052786 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 308/598] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:39:43 2022] GET /identity/v3/projects/service => generated 88 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:44.052786 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe9a8c3-215f-41e5-909b-66fb1ee25a24 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:44.093464 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ceadca5-d66a-49ef-bca3-690789bad282 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:44.094687 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ceadca5-d66a-49ef-bca3-690789bad282 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:44.094687 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ceadca5-d66a-49ef-bca3-690789bad282 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:44.094987 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ceadca5-d66a-49ef-bca3-690789bad282 None admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:44.095902 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ceadca5-d66a-49ef-bca3-690789bad282 None admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:44.096167 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ceadca5-d66a-49ef-bca3-690789bad282 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:44.097810 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ceadca5-d66a-49ef-bca3-690789bad282 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:44.119391 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 291/599] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:39:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:44.121077 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ceadca5-d66a-49ef-bca3-690789bad282 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:44.149284 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:44.190336 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe9a8c3-215f-41e5-909b-66fb1ee25a24 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:44.235383 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5219aa41-8ac8-4d59-875d-dd4dce6dd7e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:44.236010 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5219aa41-8ac8-4d59-875d-dd4dce6dd7e6 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:44.236284 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5219aa41-8ac8-4d59-875d-dd4dce6dd7e6 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:44.236549 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5219aa41-8ac8-4d59-875d-dd4dce6dd7e6 None admin] PATH_INFO: `/v3/projects/service` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:44.244216 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5219aa41-8ac8-4d59-875d-dd4dce6dd7e6 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:44.245667 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5219aa41-8ac8-4d59-875d-dd4dce6dd7e6 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:44.256931 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-5219aa41-8ac8-4d59-875d-dd4dce6dd7e6 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:39:44.256931 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:44.256931 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:44.256931 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:44.256931 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:44.256931 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:44.256931 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:44.256931 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:44.256931 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:44.256931 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:44.256931 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:44.256931 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:44.256931 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 11 00:39:44.256931 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 11 00:39:44.256931 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 11 00:39:44.258187 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 11 00:39:44.258187 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:44.258187 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:44.258187 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:44.258187 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:44.258187 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:44.258187 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:44.258187 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:44.258187 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:44.258187 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:44.258187 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:44.258187 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:44.258187 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:44.258187 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:44.258187 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:44.258187 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:44.258187 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:44.259118 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 11 00:39:44.259118 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 11 00:39:44.259118 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 11 00:39:44.259118 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 11 00:39:44.259118 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 11 00:39:44.259118 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 11 00:39:44.259118 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:39:44.259118 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:44.259118 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 309/600] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:39:44 2022] GET /identity/v3/projects/service => generated 88 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:44.259671 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ceadca5-d66a-49ef-bca3-690789bad282 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:44.259753 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5219aa41-8ac8-4d59-875d-dd4dce6dd7e6 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:44.297448 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec2d789-de1a-4159-9c79-10a9a5d8d8c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:44.297968 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec2d789-de1a-4159-9c79-10a9a5d8d8c5 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:44.298240 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec2d789-de1a-4159-9c79-10a9a5d8d8c5 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:44.298506 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec2d789-de1a-4159-9c79-10a9a5d8d8c5 None admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:44.299212 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dec2d789-de1a-4159-9c79-10a9a5d8d8c5 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:44.299507 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dec2d789-de1a-4159-9c79-10a9a5d8d8c5 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:44.300927 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dec2d789-de1a-4159-9c79-10a9a5d8d8c5 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:44.318002 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 292/601] 10.222.0.27 () {56 vars in 1148 bytes} [Sun Sep 11 00:39:44 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:44.318002 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a115c33-d785-4421-a4e8-20ffe1bc22fa None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:44.318002 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a115c33-d785-4421-a4e8-20ffe1bc22fa None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:44.318002 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a115c33-d785-4421-a4e8-20ffe1bc22fa None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:44.320646 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 293/602] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:39:44 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:39:44.321356 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a115c33-d785-4421-a4e8-20ffe1bc22fa None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:44.323418 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:44.374908 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5219aa41-8ac8-4d59-875d-dd4dce6dd7e6 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:44.402926 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:44.413942 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb81eeb-57fc-4352-93cf-3cfcb72b79bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:44.415171 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb81eeb-57fc-4352-93cf-3cfcb72b79bb None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:44.415450 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb81eeb-57fc-4352-93cf-3cfcb72b79bb None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:44.416184 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb81eeb-57fc-4352-93cf-3cfcb72b79bb None admin] PATH_INFO: `/v3/projects/service` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:44.423161 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fb81eeb-57fc-4352-93cf-3cfcb72b79bb None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:44.424639 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fb81eeb-57fc-4352-93cf-3cfcb72b79bb None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:44.428624 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a115c33-d785-4421-a4e8-20ffe1bc22fa None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:44.438254 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-4fb81eeb-57fc-4352-93cf-3cfcb72b79bb None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:39:44.438254 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:44.438254 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:44.438254 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:44.438254 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:44.438254 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:44.438254 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:44.438254 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:44.438254 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:44.438254 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:44.438254 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:44.438254 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:44.438254 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 11 00:39:44.438254 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 11 00:39:44.438254 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 11 00:39:44.439174 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 11 00:39:44.439174 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:44.439174 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:44.439174 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:44.439174 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:44.439174 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:44.439174 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:44.439174 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:44.439174 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:44.439174 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:44.439174 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:44.439174 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:44.439174 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:44.439174 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:44.439174 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:44.439174 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:44.439174 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:44.440098 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 11 00:39:44.440098 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 11 00:39:44.440098 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 11 00:39:44.440098 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 11 00:39:44.440098 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 11 00:39:44.440098 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 11 00:39:44.440098 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:39:44.440098 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:44.440098 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 310/603] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:39:44 2022] GET /identity/v3/projects/service => generated 88 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:44.441187 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb81eeb-57fc-4352-93cf-3cfcb72b79bb None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:44.473157 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27b7cc06-c7ea-42d8-a27f-06bfd1c6f2ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:44.474629 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b7cc06-c7ea-42d8-a27f-06bfd1c6f2ff None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:44.475171 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b7cc06-c7ea-42d8-a27f-06bfd1c6f2ff None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:44.475506 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b7cc06-c7ea-42d8-a27f-06bfd1c6f2ff None admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:44.476598 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27b7cc06-c7ea-42d8-a27f-06bfd1c6f2ff None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:44.477013 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27b7cc06-c7ea-42d8-a27f-06bfd1c6f2ff None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:44.478469 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27b7cc06-c7ea-42d8-a27f-06bfd1c6f2ff None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:44.490113 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 294/604] 10.222.0.27 () {56 vars in 1148 bytes} [Sun Sep 11 00:39:44 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:44.491335 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee0dc28-41c4-43e7-8a85-74807a2752dd None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:44.491787 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee0dc28-41c4-43e7-8a85-74807a2752dd None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:44.492277 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee0dc28-41c4-43e7-8a85-74807a2752dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:44.510500 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-dee0dc28-41c4-43e7-8a85-74807a2752dd None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:39:44.511719 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-dee0dc28-41c4-43e7-8a85-74807a2752dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:39:44.556075 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb81eeb-57fc-4352-93cf-3cfcb72b79bb None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:44.571297 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:44.586972 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 295/605] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:39:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:39:44.586972 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee0dc28-41c4-43e7-8a85-74807a2752dd None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:44.615119 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a2fe3d-6a0a-46f7-b1ce-c058ec59cada None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:44.616257 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a2fe3d-6a0a-46f7-b1ce-c058ec59cada None admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:44.617119 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a2fe3d-6a0a-46f7-b1ce-c058ec59cada None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:44.617549 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a2fe3d-6a0a-46f7-b1ce-c058ec59cada None admin] PATH_INFO: `/v3/projects/b504a5058bc6474c8cc077dfeec16799/users/22374eaff0834245b88506dcd7b24ede/roles/0390f926e7ef4715ae18697033bde90e` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:44.636615 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37a2fe3d-6a0a-46f7-b1ce-c058ec59cada None admin] RBAC: Authorizing `identity:create_grant(project_id=b504a5058bc6474c8cc077dfeec16799, user_id=22374eaff0834245b88506dcd7b24ede, role_id=0390f926e7ef4715ae18697033bde90e)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:44.638691 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37a2fe3d-6a0a-46f7-b1ce-c058ec59cada None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:44.662549 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 311/606] 10.222.0.27 () {58 vars in 1439 bytes} [Sun Sep 11 00:39:44 2022] PUT /identity/v3/projects/b504a5058bc6474c8cc077dfeec16799/users/22374eaff0834245b88506dcd7b24ede/roles/0390f926e7ef4715ae18697033bde90e => generated 0 bytes in 217 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:39:44.664919 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:44.666471 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a2fe3d-6a0a-46f7-b1ce-c058ec59cada None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:44.728321 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee0dc28-41c4-43e7-8a85-74807a2752dd None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:44.812283 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c2f4bbd-859b-4ebe-a0f3-cf1beef07bc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:44.812822 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2f4bbd-859b-4ebe-a0f3-cf1beef07bc5 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:44.813101 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2f4bbd-859b-4ebe-a0f3-cf1beef07bc5 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:44.813374 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2f4bbd-859b-4ebe-a0f3-cf1beef07bc5 None admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:44.814072 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c2f4bbd-859b-4ebe-a0f3-cf1beef07bc5 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:44.814378 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c2f4bbd-859b-4ebe-a0f3-cf1beef07bc5 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:44.815920 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c2f4bbd-859b-4ebe-a0f3-cf1beef07bc5 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:44.853728 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 296/607] 10.222.0.27 () {56 vars in 1148 bytes} [Sun Sep 11 00:39:44 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 270 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:44.861892 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c2f4bbd-859b-4ebe-a0f3-cf1beef07bc5 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:44.890134 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a2fe3d-6a0a-46f7-b1ce-c058ec59cada None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:44.932598 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1361d937-5690-4231-b634-68d3007ba8aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:44.933429 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1361d937-5690-4231-b634-68d3007ba8aa None admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:44.933708 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1361d937-5690-4231-b634-68d3007ba8aa None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:44.933980 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1361d937-5690-4231-b634-68d3007ba8aa None admin] PATH_INFO: `/v3/projects/b504a5058bc6474c8cc077dfeec16799/users/a5f7b13283cb4ebfb25160aa6ad0e2f4/roles/0390f926e7ef4715ae18697033bde90e` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:44.953805 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1361d937-5690-4231-b634-68d3007ba8aa None admin] RBAC: Authorizing `identity:create_grant(project_id=b504a5058bc6474c8cc077dfeec16799, user_id=a5f7b13283cb4ebfb25160aa6ad0e2f4, role_id=0390f926e7ef4715ae18697033bde90e)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:44.955726 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1361d937-5690-4231-b634-68d3007ba8aa None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:44.975014 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 312/608] 10.222.0.27 () {58 vars in 1439 bytes} [Sun Sep 11 00:39:44 2022] PUT /identity/v3/projects/b504a5058bc6474c8cc077dfeec16799/users/a5f7b13283cb4ebfb25160aa6ad0e2f4/roles/0390f926e7ef4715ae18697033bde90e => generated 0 bytes in 311 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:39:44.983504 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1361d937-5690-4231-b634-68d3007ba8aa None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:45.009562 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c2f4bbd-859b-4ebe-a0f3-cf1beef07bc5 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:45.014586 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:45.061488 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:45.087157 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b142add1-0687-482d-b4c1-448b578423cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:45.087761 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b142add1-0687-482d-b4c1-448b578423cf None admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:45.088045 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b142add1-0687-482d-b4c1-448b578423cf None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:45.088318 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b142add1-0687-482d-b4c1-448b578423cf None admin] PATH_INFO: `/v3/projects/b504a5058bc6474c8cc077dfeec16799/users/06b1c911731f4267a5f37e91d948e7e9/roles/0390f926e7ef4715ae18697033bde90e` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:45.121190 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b142add1-0687-482d-b4c1-448b578423cf None admin] RBAC: Authorizing `identity:create_grant(project_id=b504a5058bc6474c8cc077dfeec16799, user_id=06b1c911731f4267a5f37e91d948e7e9, role_id=0390f926e7ef4715ae18697033bde90e)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:45.123931 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b142add1-0687-482d-b4c1-448b578423cf None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:45.147485 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 297/609] 10.222.0.27 () {58 vars in 1439 bytes} [Sun Sep 11 00:39:44 2022] PUT /identity/v3/projects/b504a5058bc6474c8cc077dfeec16799/users/06b1c911731f4267a5f37e91d948e7e9/roles/0390f926e7ef4715ae18697033bde90e => generated 0 bytes in 286 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:39:45.206957 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1361d937-5690-4231-b634-68d3007ba8aa None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:45.256199 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70b4f7d5-9305-45ad-a41a-85d6a7590120 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:45.256913 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b4f7d5-9305-45ad-a41a-85d6a7590120 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:45.257736 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b4f7d5-9305-45ad-a41a-85d6a7590120 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:45.258012 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b4f7d5-9305-45ad-a41a-85d6a7590120 None admin] PATH_INFO: `/v3/roles/service` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:45.275378 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70b4f7d5-9305-45ad-a41a-85d6a7590120 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:39:45.280560 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-70b4f7d5-9305-45ad-a41a-85d6a7590120 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 11 00:39:45.280560 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:45.280560 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:45.280560 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:45.280560 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:45.280560 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:45.280560 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:45.280560 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:45.280560 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:45.280560 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:45.280560 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:45.280560 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:45.280560 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:39:45.280560 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:39:45.280560 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:39:45.280560 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:39:45.281737 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:39:45.281737 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:39:45.281737 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:39:45.281737 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:39:45.281737 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:45.281737 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:45.281737 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:45.281737 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:45.281737 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:45.281737 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:45.281737 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:45.281737 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:45.281737 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:45.281737 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:45.281737 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:45.281737 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:45.282592 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:45.282592 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:45.282592 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:45.282592 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:45.282592 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:39:45.282592 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:39:45.282592 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:39:45.282592 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:39:45.282592 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:39:45.282592 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:39:45.282592 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 11 00:39:45.282592 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:45.282592 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 313/610] 10.222.0.27 () {56 vars in 1102 bytes} [Sun Sep 11 00:39:44 2022] GET /identity/v3/roles/service => generated 85 bytes in 306 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:45.289228 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b142add1-0687-482d-b4c1-448b578423cf None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:45.415810 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b142add1-0687-482d-b4c1-448b578423cf None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:45.462410 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c7d906-bfda-437d-b137-81e42b68dcdd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:45.463061 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c7d906-bfda-437d-b137-81e42b68dcdd None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:45.463336 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c7d906-bfda-437d-b137-81e42b68dcdd None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:45.463605 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c7d906-bfda-437d-b137-81e42b68dcdd None admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:45.464302 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7c7d906-bfda-437d-b137-81e42b68dcdd None admin] RBAC: Adding query filter params (name=service) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:45.464599 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7c7d906-bfda-437d-b137-81e42b68dcdd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:45.466005 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7c7d906-bfda-437d-b137-81e42b68dcdd None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:45.475637 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 298/611] 10.222.0.27 () {56 vars in 1103 bytes} [Sun Sep 11 00:39:45 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:45.482757 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70b4f7d5-9305-45ad-a41a-85d6a7590120 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:45.505682 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bec0a0a-cb94-47ee-b241-cfa0fe0310e4 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:45.506004 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bec0a0a-cb94-47ee-b241-cfa0fe0310e4 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:45.506237 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bec0a0a-cb94-47ee-b241-cfa0fe0310e4 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:45.507956 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 299/612] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:39:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:39:45.513113 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90584de4-a77b-4287-8466-e54e9645354c None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:45.513381 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90584de4-a77b-4287-8466-e54e9645354c None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:45.513648 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90584de4-a77b-4287-8466-e54e9645354c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:45.533820 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-90584de4-a77b-4287-8466-e54e9645354c None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:39:45.535047 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-90584de4-a77b-4287-8466-e54e9645354c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:39:45.569029 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:45.571944 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:45.606089 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70b4f7d5-9305-45ad-a41a-85d6a7590120 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:45.611439 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 300/613] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:39:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 11 00:39:45.643401 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b4dfa14-ea72-4387-bb76-892bc28953c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:45.644430 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4dfa14-ea72-4387-bb76-892bc28953c6 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:45.644914 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4dfa14-ea72-4387-bb76-892bc28953c6 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:45.645383 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4dfa14-ea72-4387-bb76-892bc28953c6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:45.656920 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b4dfa14-ea72-4387-bb76-892bc28953c6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:45.658735 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b4dfa14-ea72-4387-bb76-892bc28953c6 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:45.675190 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-5b4dfa14-ea72-4387-bb76-892bc28953c6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:45.675190 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:45.675190 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:45.675190 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:45.675190 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:45.675190 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:45.675190 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:45.675190 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:45.675190 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:45.675190 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:45.675190 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:45.675190 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:45.675190 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:39:45.675190 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:39:45.675190 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:39:45.675190 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:39:45.676152 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:45.676152 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:45.676152 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:45.676152 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:45.676152 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:45.676152 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:45.676152 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:45.676152 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:45.676152 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:45.676152 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:45.676152 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:45.676152 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:45.676152 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:45.676152 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:45.676152 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:45.676152 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:45.676152 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:39:45.677070 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:39:45.677070 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:45.677070 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:45.679970 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 314/614] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:39:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:45.685946 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90584de4-a77b-4287-8466-e54e9645354c None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:45.829030 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90584de4-a77b-4287-8466-e54e9645354c None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:45.880886 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b729f6bc-1a1f-4796-97cc-97a42dacec8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:45.881499 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b729f6bc-1a1f-4796-97cc-97a42dacec8e None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:45.881780 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b729f6bc-1a1f-4796-97cc-97a42dacec8e None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:45.882116 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b729f6bc-1a1f-4796-97cc-97a42dacec8e None admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:45.882849 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b729f6bc-1a1f-4796-97cc-97a42dacec8e None admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:45.883159 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b729f6bc-1a1f-4796-97cc-97a42dacec8e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:45.884539 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b729f6bc-1a1f-4796-97cc-97a42dacec8e None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:45.899931 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 301/615] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:39:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:45.905332 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b4dfa14-ea72-4387-bb76-892bc28953c6 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:46.001179 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b729f6bc-1a1f-4796-97cc-97a42dacec8e None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:46.025795 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b4dfa14-ea72-4387-bb76-892bc28953c6 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:46.051978 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:46.056910 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:46.065683 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ac0c69-a868-4423-961f-8d53de33e690 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:46.066636 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ac0c69-a868-4423-961f-8d53de33e690 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:46.067085 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ac0c69-a868-4423-961f-8d53de33e690 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:46.067512 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ac0c69-a868-4423-961f-8d53de33e690 None admin] PATH_INFO: `/v3/users/swift` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:46.081913 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22ac0c69-a868-4423-961f-8d53de33e690 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:46.083696 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22ac0c69-a868-4423-961f-8d53de33e690 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:46.096845 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-22ac0c69-a868-4423-961f-8d53de33e690 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 11 00:39:46.096845 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:46.096845 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:46.096845 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:46.096845 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:46.096845 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:46.096845 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:46.096845 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:46.096845 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:46.096845 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:46.096845 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:46.096845 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:46.096845 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 11 00:39:46.096845 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 11 00:39:46.096845 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 11 00:39:46.096845 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 11 00:39:46.097993 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:46.097993 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:46.097993 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 11 00:39:46.097993 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:39:46.097993 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 11 00:39:46.097993 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:39:46.097993 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:46.097993 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:46.097993 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:46.097993 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:46.097993 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:46.097993 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:46.097993 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:46.097993 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:46.097993 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:46.097993 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:46.097993 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:46.097993 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:46.098963 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:46.098963 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:46.098963 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 11 00:39:46.098963 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 11 00:39:46.098963 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 11 00:39:46.098963 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 11 00:39:46.098963 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 11 00:39:46.098963 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 11 00:39:46.098963 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 11 00:39:46.098963 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:46.104930 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 315/616] 10.222.0.27 () {56 vars in 1096 bytes} [Sun Sep 11 00:39:45 2022] GET /identity/v3/users/swift => generated 83 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:46.110597 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ac0c69-a868-4423-961f-8d53de33e690 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:46.211495 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b729f6bc-1a1f-4796-97cc-97a42dacec8e None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:46.221197 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ac0c69-a868-4423-961f-8d53de33e690 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:46.260003 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23fb8f05-a3da-4338-be81-0a8e5d4f2c71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:46.261020 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23fb8f05-a3da-4338-be81-0a8e5d4f2c71 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:46.261496 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23fb8f05-a3da-4338-be81-0a8e5d4f2c71 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:46.261982 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23fb8f05-a3da-4338-be81-0a8e5d4f2c71 None admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:46.262952 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23fb8f05-a3da-4338-be81-0a8e5d4f2c71 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:46.263485 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23fb8f05-a3da-4338-be81-0a8e5d4f2c71 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:46.265159 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23fb8f05-a3da-4338-be81-0a8e5d4f2c71 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:46.286166 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 316/617] 10.222.0.27 () {56 vars in 1135 bytes} [Sun Sep 11 00:39:46 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:39:46.290847 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a5e2d6-b425-42fa-98c4-2d8e44229b4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:46.290847 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a5e2d6-b425-42fa-98c4-2d8e44229b4e None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:46.290847 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a5e2d6-b425-42fa-98c4-2d8e44229b4e None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:46.290847 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a5e2d6-b425-42fa-98c4-2d8e44229b4e None admin] PATH_INFO: `/v3/roles/service` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:46.293405 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23fb8f05-a3da-4338-be81-0a8e5d4f2c71 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:46.313079 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99a5e2d6-b425-42fa-98c4-2d8e44229b4e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:39:46.318734 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-99a5e2d6-b425-42fa-98c4-2d8e44229b4e None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 11 00:39:46.318734 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:46.318734 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:46.318734 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:46.318734 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:46.318734 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:46.318734 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:46.318734 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:46.318734 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:46.318734 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:46.318734 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:46.318734 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:46.318734 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:39:46.318734 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:39:46.318734 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:39:46.318734 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:39:46.319890 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:39:46.319890 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:39:46.319890 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:39:46.319890 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:39:46.319890 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:46.319890 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:46.319890 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:46.319890 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:46.319890 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:46.319890 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:46.319890 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:46.319890 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:46.319890 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:46.319890 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:46.319890 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:46.319890 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:46.321190 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:46.321190 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:46.321190 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:46.321190 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:46.321190 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:39:46.321190 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:39:46.321190 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:39:46.321190 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:39:46.321190 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:39:46.321190 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:39:46.321190 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 11 00:39:46.321190 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:46.327339 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 302/618] 10.222.0.27 () {56 vars in 1102 bytes} [Sun Sep 11 00:39:46 2022] GET /identity/v3/roles/service => generated 85 bytes in 326 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:46.332516 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a5e2d6-b425-42fa-98c4-2d8e44229b4e None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:46.345476 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:46.407383 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23fb8f05-a3da-4338-be81-0a8e5d4f2c71 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:46.445986 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa108e42-4d66-409c-a6c7-2ad8ed12a42f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:46.447020 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa108e42-4d66-409c-a6c7-2ad8ed12a42f None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:46.447498 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa108e42-4d66-409c-a6c7-2ad8ed12a42f None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:46.447963 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa108e42-4d66-409c-a6c7-2ad8ed12a42f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:46.459108 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa108e42-4d66-409c-a6c7-2ad8ed12a42f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:46.461942 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa108e42-4d66-409c-a6c7-2ad8ed12a42f None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:46.477822 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-fa108e42-4d66-409c-a6c7-2ad8ed12a42f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:46.477822 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:46.477822 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:46.477822 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:46.477822 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:46.477822 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:46.477822 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:46.477822 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:46.477822 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:46.477822 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:46.477822 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:46.477822 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:46.477822 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:39:46.477822 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:39:46.477822 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:39:46.477822 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:39:46.479004 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:46.479004 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:46.479004 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:46.479004 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:46.479004 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:46.479004 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:46.479004 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:46.479004 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:46.479004 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:46.479004 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:46.479004 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:46.479004 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:46.479004 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:46.479004 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:46.479004 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:46.479004 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:46.479004 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:39:46.479892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:39:46.479892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:46.479892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:46.481933 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 317/619] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:39:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:46.487530 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa108e42-4d66-409c-a6c7-2ad8ed12a42f None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:46.502952 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:46.528407 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a5e2d6-b425-42fa-98c4-2d8e44229b4e None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:46.587267 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:46.595254 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa108e42-4d66-409c-a6c7-2ad8ed12a42f None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:46.603425 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa5eb10-5984-4ac8-87a5-fc06ac1f2622 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:46.604158 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa5eb10-5984-4ac8-87a5-fc06ac1f2622 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:46.604386 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa5eb10-5984-4ac8-87a5-fc06ac1f2622 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:46.604569 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa5eb10-5984-4ac8-87a5-fc06ac1f2622 None admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:46.605249 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fa5eb10-5984-4ac8-87a5-fc06ac1f2622 None admin] RBAC: Adding query filter params (name=service) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:46.605550 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fa5eb10-5984-4ac8-87a5-fc06ac1f2622 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:46.611261 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fa5eb10-5984-4ac8-87a5-fc06ac1f2622 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:46.629576 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 303/620] 10.222.0.27 () {56 vars in 1103 bytes} [Sun Sep 11 00:39:46 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 297 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:39:46.634771 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ca0296-8e60-404b-a16e-49f8a0b43408 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:46.635680 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ca0296-8e60-404b-a16e-49f8a0b43408 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:46.636155 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ca0296-8e60-404b-a16e-49f8a0b43408 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:46.636261 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa5eb10-5984-4ac8-87a5-fc06ac1f2622 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:46.636755 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ca0296-8e60-404b-a16e-49f8a0b43408 None admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:46.637675 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4ca0296-8e60-404b-a16e-49f8a0b43408 None admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:46.638179 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4ca0296-8e60-404b-a16e-49f8a0b43408 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:46.639748 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4ca0296-8e60-404b-a16e-49f8a0b43408 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:46.653088 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 318/621] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:39:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:46.658766 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ca0296-8e60-404b-a16e-49f8a0b43408 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:46.763581 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ca0296-8e60-404b-a16e-49f8a0b43408 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:46.800028 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00edc23e-28e2-4d46-96bf-955badb42e2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:46.801036 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00edc23e-28e2-4d46-96bf-955badb42e2c None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:46.801511 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00edc23e-28e2-4d46-96bf-955badb42e2c None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:46.801974 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00edc23e-28e2-4d46-96bf-955badb42e2c None admin] PATH_INFO: `/v3/projects/service` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:46.809589 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00edc23e-28e2-4d46-96bf-955badb42e2c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:46.811327 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00edc23e-28e2-4d46-96bf-955badb42e2c None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:46.824535 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-00edc23e-28e2-4d46-96bf-955badb42e2c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:39:46.824535 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:46.824535 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:46.824535 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:46.824535 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:46.824535 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:46.824535 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:46.824535 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:46.824535 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:46.824535 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:46.824535 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:46.824535 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:46.824535 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 11 00:39:46.824535 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 11 00:39:46.824535 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 11 00:39:46.825447 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 11 00:39:46.825447 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:46.825447 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:46.825447 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:46.825447 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:46.825447 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:46.825447 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:46.825447 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:46.825447 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:46.825447 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:46.825447 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:46.825447 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:46.825447 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:46.825447 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:46.825447 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:46.825447 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:46.825447 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:46.826343 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 11 00:39:46.826343 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 11 00:39:46.826343 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 11 00:39:46.826343 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 11 00:39:46.826343 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 11 00:39:46.826343 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 11 00:39:46.826343 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:39:46.826343 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:46.828739 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 319/622] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:39:46 2022] GET /identity/v3/projects/service => generated 88 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:39:46.833474 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa5eb10-5984-4ac8-87a5-fc06ac1f2622 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:46.834572 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00edc23e-28e2-4d46-96bf-955badb42e2c None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:46.867398 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:46.897093 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd859ede-5a86-4c07-8061-5b8ccd17a0ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:46.898406 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd859ede-5a86-4c07-8061-5b8ccd17a0ad None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:46.899084 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd859ede-5a86-4c07-8061-5b8ccd17a0ad None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:46.901164 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd859ede-5a86-4c07-8061-5b8ccd17a0ad None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:46.925754 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd859ede-5a86-4c07-8061-5b8ccd17a0ad None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:46.928745 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd859ede-5a86-4c07-8061-5b8ccd17a0ad None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:46.957848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-cd859ede-5a86-4c07-8061-5b8ccd17a0ad None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:46.957848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:46.957848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:46.957848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:46.957848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:46.957848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:46.957848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:46.957848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:46.957848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:46.957848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:46.957848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:46.957848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:46.957848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:39:46.957848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:39:46.957848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:39:46.957848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:39:46.958883 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:46.958883 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:46.958883 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:46.958883 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:46.958883 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:46.958883 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:46.958883 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:46.958883 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:46.958883 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:46.958883 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:46.958883 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:46.958883 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:46.958883 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:46.958883 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:46.958883 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:46.958883 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:46.958883 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:39:46.959840 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:39:46.959840 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:46.959840 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:46.966499 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 304/623] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:39:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 331 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:39:46.972790 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd859ede-5a86-4c07-8061-5b8ccd17a0ad None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:46.977205 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:47.018811 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00edc23e-28e2-4d46-96bf-955badb42e2c None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:47.070497 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ea29a48-1545-40bd-8d00-baed3e1667bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:47.071162 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea29a48-1545-40bd-8d00-baed3e1667bd None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:47.071441 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea29a48-1545-40bd-8d00-baed3e1667bd None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:47.071708 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea29a48-1545-40bd-8d00-baed3e1667bd None admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:47.072398 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ea29a48-1545-40bd-8d00-baed3e1667bd None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:47.072700 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ea29a48-1545-40bd-8d00-baed3e1667bd None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:47.074124 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ea29a48-1545-40bd-8d00-baed3e1667bd None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:47.103071 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 320/624] 10.222.0.27 () {56 vars in 1148 bytes} [Sun Sep 11 00:39:46 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:47.111474 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ea29a48-1545-40bd-8d00-baed3e1667bd None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:47.173129 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd859ede-5a86-4c07-8061-5b8ccd17a0ad None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:47.237890 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fbc68db-a1cb-4ed1-a5b0-ed60e457206e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:47.239404 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fbc68db-a1cb-4ed1-a5b0-ed60e457206e None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:47.239675 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fbc68db-a1cb-4ed1-a5b0-ed60e457206e None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:47.239939 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fbc68db-a1cb-4ed1-a5b0-ed60e457206e None admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:47.240585 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fbc68db-a1cb-4ed1-a5b0-ed60e457206e None admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:47.243211 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fbc68db-a1cb-4ed1-a5b0-ed60e457206e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:47.244634 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fbc68db-a1cb-4ed1-a5b0-ed60e457206e None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:47.259943 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:47.269418 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 305/625] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:39:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 297 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:47.277469 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fbc68db-a1cb-4ed1-a5b0-ed60e457206e None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:47.295258 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ea29a48-1545-40bd-8d00-baed3e1667bd None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:47.341721 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d56efc36-0754-47f3-910a-c247f1f73fae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:47.342743 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d56efc36-0754-47f3-910a-c247f1f73fae None admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:47.343841 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d56efc36-0754-47f3-910a-c247f1f73fae None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:47.345053 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d56efc36-0754-47f3-910a-c247f1f73fae None admin] PATH_INFO: `/v3/projects/b504a5058bc6474c8cc077dfeec16799/users/bcb73dec73464a3abc4b774abb4f416f/roles/0390f926e7ef4715ae18697033bde90e` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:47.374711 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d56efc36-0754-47f3-910a-c247f1f73fae None admin] RBAC: Authorizing `identity:create_grant(project_id=b504a5058bc6474c8cc077dfeec16799, user_id=bcb73dec73464a3abc4b774abb4f416f, role_id=0390f926e7ef4715ae18697033bde90e)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:47.377343 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d56efc36-0754-47f3-910a-c247f1f73fae None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:47.401283 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 321/626] 10.222.0.27 () {58 vars in 1439 bytes} [Sun Sep 11 00:39:47 2022] PUT /identity/v3/projects/b504a5058bc6474c8cc077dfeec16799/users/bcb73dec73464a3abc4b774abb4f416f/roles/0390f926e7ef4715ae18697033bde90e => generated 0 bytes in 290 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:39:47.425429 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:47.457209 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fbc68db-a1cb-4ed1-a5b0-ed60e457206e None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:47.512281 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c889ac-e635-4928-b029-af955af92f51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:47.513381 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c889ac-e635-4928-b029-af955af92f51 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:47.513874 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c889ac-e635-4928-b029-af955af92f51 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:47.514407 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c889ac-e635-4928-b029-af955af92f51 None admin] PATH_INFO: `/v3/users/nova` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:47.524052 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9c889ac-e635-4928-b029-af955af92f51 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:47.525976 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9c889ac-e635-4928-b029-af955af92f51 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:47.540148 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-b9c889ac-e635-4928-b029-af955af92f51 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 11 00:39:47.540148 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:47.540148 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:47.540148 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:47.540148 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:47.540148 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:47.540148 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:47.540148 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:47.540148 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:47.540148 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:47.540148 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:47.540148 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:47.540148 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 11 00:39:47.540148 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 11 00:39:47.540148 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 11 00:39:47.540148 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 11 00:39:47.541206 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:47.541206 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:47.541206 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 11 00:39:47.541206 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:39:47.541206 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 11 00:39:47.541206 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:39:47.541206 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:47.541206 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:47.541206 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:47.541206 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:47.541206 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:47.541206 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:47.541206 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:47.541206 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:47.541206 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:47.541206 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:47.541206 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:47.541206 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:47.542239 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:47.542239 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:47.542239 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 11 00:39:47.542239 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 11 00:39:47.542239 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 11 00:39:47.542239 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 11 00:39:47.542239 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 11 00:39:47.542239 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 11 00:39:47.542239 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 11 00:39:47.542239 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:47.544989 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 306/627] 10.222.0.27 () {56 vars in 1093 bytes} [Sun Sep 11 00:39:47 2022] GET /identity/v3/users/nova => generated 82 bytes in 270 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:47.551539 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d56efc36-0754-47f3-910a-c247f1f73fae None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:47.611905 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21aa97f4-d829-42f4-b18a-6cded30bcc22 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:47.612103 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21aa97f4-d829-42f4-b18a-6cded30bcc22 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:47.612360 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21aa97f4-d829-42f4-b18a-6cded30bcc22 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:47.613966 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 307/628] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:39:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:39:47.619451 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62966136-0b55-4031-a8ca-07685cbd7b62 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:47.619754 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62966136-0b55-4031-a8ca-07685cbd7b62 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:47.619991 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62966136-0b55-4031-a8ca-07685cbd7b62 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:47.639767 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-62966136-0b55-4031-a8ca-07685cbd7b62 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:39:47.641001 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-62966136-0b55-4031-a8ca-07685cbd7b62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:39:47.675383 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d56efc36-0754-47f3-910a-c247f1f73fae None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:47.712861 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:47.732370 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5caddb28-d750-4296-88e9-28d533570e8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:47.733330 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5caddb28-d750-4296-88e9-28d533570e8d None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:47.733877 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5caddb28-d750-4296-88e9-28d533570e8d None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:47.734150 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5caddb28-d750-4296-88e9-28d533570e8d None admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:47.735372 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5caddb28-d750-4296-88e9-28d533570e8d None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:47.735372 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5caddb28-d750-4296-88e9-28d533570e8d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:47.737396 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5caddb28-d750-4296-88e9-28d533570e8d None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:47.744179 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 308/629] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:39:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 11 00:39:47.744880 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0cbb3ae-3af0-4f33-a612-8557d79036c9 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:47.745161 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0cbb3ae-3af0-4f33-a612-8557d79036c9 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:47.745439 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0cbb3ae-3af0-4f33-a612-8557d79036c9 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:47.746855 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 309/630] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:39:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:39:47.756304 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06dc9d0f-06a5-4402-b659-2a5e250154c1 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:47.756584 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06dc9d0f-06a5-4402-b659-2a5e250154c1 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:47.756851 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06dc9d0f-06a5-4402-b659-2a5e250154c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:47.770461 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 322/631] 10.222.0.27 () {56 vars in 1133 bytes} [Sun Sep 11 00:39:47 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:47.778979 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-06dc9d0f-06a5-4402-b659-2a5e250154c1 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:39:47.780101 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5caddb28-d750-4296-88e9-28d533570e8d None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:47.780239 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-06dc9d0f-06a5-4402-b659-2a5e250154c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:39:47.848071 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:47.890585 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 310/632] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:39:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:39:47.915475 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5caddb28-d750-4296-88e9-28d533570e8d None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:47.955502 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9364f2dc-dfe3-4584-aa26-252f7c8cebed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:47.956517 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9364f2dc-dfe3-4584-aa26-252f7c8cebed None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:47.957160 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9364f2dc-dfe3-4584-aa26-252f7c8cebed None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:47.957662 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9364f2dc-dfe3-4584-aa26-252f7c8cebed None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:47.970812 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c28d8c3-9d4f-4c58-a1de-e9aa3760f877 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:47.970812 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c28d8c3-9d4f-4c58-a1de-e9aa3760f877 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:47.970812 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c28d8c3-9d4f-4c58-a1de-e9aa3760f877 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:47.971603 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 311/633] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:39:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 11 00:39:47.977477 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-778e31e7-9024-4ec5-825c-5cd6e8ff704a None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:47.977757 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-778e31e7-9024-4ec5-825c-5cd6e8ff704a None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:47.978026 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-778e31e7-9024-4ec5-825c-5cd6e8ff704a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:48.002700 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-778e31e7-9024-4ec5-825c-5cd6e8ff704a None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:39:48.003955 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-778e31e7-9024-4ec5-825c-5cd6e8ff704a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:39:48.037824 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:48.084752 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9364f2dc-dfe3-4584-aa26-252f7c8cebed None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:48.086395 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9364f2dc-dfe3-4584-aa26-252f7c8cebed None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:48.112503 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 312/634] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:39:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 11 00:39:48.207377 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 323/635] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:39:47 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 428 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 11 00:39:48.210270 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-778e31e7-9024-4ec5-825c-5cd6e8ff704a None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:48.217608 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9364f2dc-dfe3-4584-aa26-252f7c8cebed None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:48.313962 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:48.332161 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-778e31e7-9024-4ec5-825c-5cd6e8ff704a None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:48.337378 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9364f2dc-dfe3-4584-aa26-252f7c8cebed None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:48.350353 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:48.379065 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79222cc9-3462-4758-9cdc-2c09648236cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:48.379548 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe29fc5a-dc64-45c0-8e73-ae3726b859a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:48.379682 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79222cc9-3462-4758-9cdc-2c09648236cd None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:48.379968 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79222cc9-3462-4758-9cdc-2c09648236cd None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:48.380243 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79222cc9-3462-4758-9cdc-2c09648236cd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:48.380961 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe29fc5a-dc64-45c0-8e73-ae3726b859a5 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:48.381435 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe29fc5a-dc64-45c0-8e73-ae3726b859a5 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:48.382777 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe29fc5a-dc64-45c0-8e73-ae3726b859a5 None admin] PATH_INFO: `/v3/roles/service` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:48.392370 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79222cc9-3462-4758-9cdc-2c09648236cd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:48.393819 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79222cc9-3462-4758-9cdc-2c09648236cd None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:48.396231 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe29fc5a-dc64-45c0-8e73-ae3726b859a5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:39:48.402667 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-fe29fc5a-dc64-45c0-8e73-ae3726b859a5 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 11 00:39:48.402667 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:48.402667 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:48.402667 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:48.402667 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:48.402667 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:48.402667 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:48.402667 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:48.402667 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:48.402667 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:48.402667 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:48.402667 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:48.402667 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:39:48.402667 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:39:48.402667 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:39:48.402667 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:39:48.403644 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:39:48.403644 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:39:48.403644 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:39:48.403644 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:39:48.403644 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:48.403644 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:48.403644 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:48.403644 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:48.403644 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:48.403644 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:48.403644 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:48.403644 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:48.403644 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:48.403644 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:48.403644 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:48.403644 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:48.404480 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:48.404480 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:48.404480 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:48.404480 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:48.404480 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:39:48.404480 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:39:48.404480 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:39:48.404480 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:39:48.404480 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:39:48.404480 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:39:48.404480 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 11 00:39:48.404480 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:48.407498 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 313/636] 10.222.0.27 () {56 vars in 1102 bytes} [Sun Sep 11 00:39:48 2022] GET /identity/v3/roles/service => generated 85 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:39:48.408545 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe29fc5a-dc64-45c0-8e73-ae3726b859a5 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:48.412635 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-79222cc9-3462-4758-9cdc-2c09648236cd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:48.412635 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:48.412635 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:48.412635 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:48.412635 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:48.412635 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:48.412635 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:48.412635 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:48.412635 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:48.412635 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:48.412635 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:48.412635 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:48.412635 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:39:48.412635 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:39:48.412635 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:39:48.412635 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:39:48.413776 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:48.413776 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:48.413776 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:48.413776 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:48.413776 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:48.413776 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:48.413776 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:48.413776 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:48.413776 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:48.413776 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:48.413776 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:48.413776 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:48.413776 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:48.413776 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:48.413776 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:48.413776 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:48.413776 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:39:48.415072 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:39:48.415072 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:48.415072 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:48.415072 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 324/637] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:39:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:39:48.415605 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79222cc9-3462-4758-9cdc-2c09648236cd None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:48.530585 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe29fc5a-dc64-45c0-8e73-ae3726b859a5 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:48.532097 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79222cc9-3462-4758-9cdc-2c09648236cd None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:48.564952 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a862fc-3a6c-4d98-a752-b19cfdf8fd3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:48.565576 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a862fc-3a6c-4d98-a752-b19cfdf8fd3a None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:48.565852 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a862fc-3a6c-4d98-a752-b19cfdf8fd3a None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:48.566118 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a862fc-3a6c-4d98-a752-b19cfdf8fd3a None admin] PATH_INFO: `/v3/roles/service` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:48.568205 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ceec764-a0e4-4a8a-b088-95fb7af63f69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:48.568942 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ceec764-a0e4-4a8a-b088-95fb7af63f69 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:48.569205 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ceec764-a0e4-4a8a-b088-95fb7af63f69 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:48.569801 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ceec764-a0e4-4a8a-b088-95fb7af63f69 None admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:48.570173 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ceec764-a0e4-4a8a-b088-95fb7af63f69 None admin] RBAC: Adding query filter params (name=service) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:48.570473 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ceec764-a0e4-4a8a-b088-95fb7af63f69 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:48.571933 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ceec764-a0e4-4a8a-b088-95fb7af63f69 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:48.573341 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:48.578135 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3a862fc-3a6c-4d98-a752-b19cfdf8fd3a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:39:48.581485 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 325/638] 10.222.0.27 () {56 vars in 1103 bytes} [Sun Sep 11 00:39:48 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:48.582467 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ceec764-a0e4-4a8a-b088-95fb7af63f69 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:48.586054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-b3a862fc-3a6c-4d98-a752-b19cfdf8fd3a None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 11 00:39:48.586054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:48.586054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:48.586054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:48.586054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:48.586054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:48.586054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:48.586054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:48.586054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:48.586054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:48.586054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:48.586054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:48.586054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:39:48.586054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:39:48.586054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:39:48.586054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:39:48.587052 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:39:48.587052 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:39:48.587052 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:39:48.587052 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:39:48.587052 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:48.587052 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:48.587052 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:48.587052 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:48.587052 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:48.587052 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:48.587052 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:48.587052 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:48.587052 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:48.587052 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:48.587052 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:48.587052 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:48.588024 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:48.588024 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:48.588024 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:48.588024 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:48.588024 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:39:48.588024 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:39:48.588024 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:39:48.588024 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:39:48.588024 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:39:48.588024 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:39:48.588024 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 11 00:39:48.588024 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:48.591077 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 314/639] 10.222.0.27 () {56 vars in 1102 bytes} [Sun Sep 11 00:39:48 2022] GET /identity/v3/roles/service => generated 85 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:48.592206 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a862fc-3a6c-4d98-a752-b19cfdf8fd3a None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:48.625531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:48.696762 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ceec764-a0e4-4a8a-b088-95fb7af63f69 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:48.697607 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a862fc-3a6c-4d98-a752-b19cfdf8fd3a None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:48.732317 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c0feb0-dcdb-4c65-b88f-6e86b2992ff2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:48.732936 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c0feb0-dcdb-4c65-b88f-6e86b2992ff2 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:48.733210 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c0feb0-dcdb-4c65-b88f-6e86b2992ff2 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:48.733483 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c0feb0-dcdb-4c65-b88f-6e86b2992ff2 None admin] PATH_INFO: `/v3/roles/service` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:48.737490 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9c6d97-b9f1-4409-8d80-34e05474c96b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:48.738194 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9c6d97-b9f1-4409-8d80-34e05474c96b None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:48.738477 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9c6d97-b9f1-4409-8d80-34e05474c96b None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:48.738612 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9c6d97-b9f1-4409-8d80-34e05474c96b None admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:48.739277 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc9c6d97-b9f1-4409-8d80-34e05474c96b None admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:48.739577 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc9c6d97-b9f1-4409-8d80-34e05474c96b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:48.740931 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc9c6d97-b9f1-4409-8d80-34e05474c96b None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:48.745154 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74c0feb0-dcdb-4c65-b88f-6e86b2992ff2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:39:48.749933 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-74c0feb0-dcdb-4c65-b88f-6e86b2992ff2 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 11 00:39:48.749933 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:48.749933 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:48.749933 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:48.749933 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:48.749933 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:48.749933 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:48.749933 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:48.749933 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:48.749933 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:48.749933 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:48.749933 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:48.749933 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:39:48.749933 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:39:48.749933 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:39:48.749933 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:39:48.751099 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:39:48.751099 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:39:48.751099 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:39:48.751099 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:39:48.751099 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:48.751099 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:48.751099 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:48.751099 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:48.751099 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:48.751099 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:48.751099 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:48.751099 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:48.751099 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:48.751099 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:48.751099 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:48.751099 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:48.752171 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:48.752171 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:48.752171 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:48.752171 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:48.752171 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:39:48.752171 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:39:48.752171 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:39:48.752171 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:39:48.752171 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:39:48.752171 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:39:48.752171 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 11 00:39:48.752171 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:48.755808 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 315/640] 10.222.0.27 () {56 vars in 1102 bytes} [Sun Sep 11 00:39:48 2022] GET /identity/v3/roles/service => generated 85 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:48.756501 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c0feb0-dcdb-4c65-b88f-6e86b2992ff2 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:48.761672 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 326/641] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:39:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:48.761672 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9c6d97-b9f1-4409-8d80-34e05474c96b None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:48.821186 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:48.854836 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:48.864617 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c0feb0-dcdb-4c65-b88f-6e86b2992ff2 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:48.869815 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9c6d97-b9f1-4409-8d80-34e05474c96b None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:48.904822 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d406b87b-27b0-4f2d-8ecd-8063469f5539 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:48.905063 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9729c22a-0dcd-4fba-9fa1-1b4e6d921b98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:48.905430 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9729c22a-0dcd-4fba-9fa1-1b4e6d921b98 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:48.905551 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d406b87b-27b0-4f2d-8ecd-8063469f5539 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:48.905751 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d406b87b-27b0-4f2d-8ecd-8063469f5539 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:48.905852 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9729c22a-0dcd-4fba-9fa1-1b4e6d921b98 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:48.905952 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d406b87b-27b0-4f2d-8ecd-8063469f5539 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:48.906028 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9729c22a-0dcd-4fba-9fa1-1b4e6d921b98 None admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:48.906636 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9729c22a-0dcd-4fba-9fa1-1b4e6d921b98 None admin] RBAC: Adding query filter params (name=service) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:48.906950 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9729c22a-0dcd-4fba-9fa1-1b4e6d921b98 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:48.908312 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9729c22a-0dcd-4fba-9fa1-1b4e6d921b98 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:48.917478 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 327/642] 10.222.0.27 () {56 vars in 1103 bytes} [Sun Sep 11 00:39:48 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:48.917830 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d406b87b-27b0-4f2d-8ecd-8063469f5539 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:48.918271 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9729c22a-0dcd-4fba-9fa1-1b4e6d921b98 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:48.919891 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d406b87b-27b0-4f2d-8ecd-8063469f5539 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:48.937645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-d406b87b-27b0-4f2d-8ecd-8063469f5539 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:48.937645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:48.937645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:48.937645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:48.937645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:48.937645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:48.937645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:48.937645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:48.937645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:48.937645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:48.937645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:48.937645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:48.937645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:39:48.937645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:39:48.937645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:39:48.937645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:39:48.938967 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:48.938967 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:48.938967 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:48.938967 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:48.938967 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:48.938967 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:48.938967 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:48.938967 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:48.938967 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:48.938967 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:48.938967 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:48.938967 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:48.938967 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:48.938967 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:48.938967 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:48.938967 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:48.938967 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:39:48.940513 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:39:48.940513 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:48.940513 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:48.940513 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 316/643] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:39:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:48.940513 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d406b87b-27b0-4f2d-8ecd-8063469f5539 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:49.039902 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9729c22a-0dcd-4fba-9fa1-1b4e6d921b98 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:49.044121 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d406b87b-27b0-4f2d-8ecd-8063469f5539 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:49.058214 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:49.077268 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f91026b4-7658-46a5-a638-a8ed4b71eac4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:49.077930 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f91026b4-7658-46a5-a638-a8ed4b71eac4 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:49.078307 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f91026b4-7658-46a5-a638-a8ed4b71eac4 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:49.078674 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f91026b4-7658-46a5-a638-a8ed4b71eac4 None admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:49.079453 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f91026b4-7658-46a5-a638-a8ed4b71eac4 None admin] RBAC: Adding query filter params (name=service) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:49.079858 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f91026b4-7658-46a5-a638-a8ed4b71eac4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:49.081344 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f91026b4-7658-46a5-a638-a8ed4b71eac4 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:49.082645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3733d60-8eef-4253-9a73-987f38b52e61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:49.083307 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3733d60-8eef-4253-9a73-987f38b52e61 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:49.083586 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3733d60-8eef-4253-9a73-987f38b52e61 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:49.084064 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3733d60-8eef-4253-9a73-987f38b52e61 None admin] PATH_INFO: `/v3/projects/service` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:49.095081 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 328/644] 10.222.0.27 () {56 vars in 1103 bytes} [Sun Sep 11 00:39:48 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:49.095081 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f91026b4-7658-46a5-a638-a8ed4b71eac4 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:49.097352 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3733d60-8eef-4253-9a73-987f38b52e61 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:49.101493 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3733d60-8eef-4253-9a73-987f38b52e61 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:49.114357 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-d3733d60-8eef-4253-9a73-987f38b52e61 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:39:49.114357 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:49.114357 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:49.114357 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:49.114357 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:49.114357 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:49.114357 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:49.114357 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:49.114357 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:49.114357 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:49.114357 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:49.114357 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:49.114357 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 11 00:39:49.114357 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 11 00:39:49.114357 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 11 00:39:49.115460 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 11 00:39:49.115460 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:49.115460 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:49.115460 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:49.115460 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:49.115460 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:49.115460 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:49.115460 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:49.115460 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:49.115460 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:49.115460 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:49.115460 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:49.115460 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:49.115460 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:49.115460 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:49.115460 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:49.115460 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:49.116925 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 11 00:39:49.116925 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 11 00:39:49.116925 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 11 00:39:49.116925 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 11 00:39:49.116925 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 11 00:39:49.116925 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 11 00:39:49.116925 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:39:49.116925 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:49.120469 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 317/645] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:39:48 2022] GET /identity/v3/projects/service => generated 88 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:49.121090 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3733d60-8eef-4253-9a73-987f38b52e61 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:49.125419 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:49.201168 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f91026b4-7658-46a5-a638-a8ed4b71eac4 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:49.226626 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3733d60-8eef-4253-9a73-987f38b52e61 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:49.239145 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a15e444b-9c8c-450f-a29c-b72983b55be1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:49.239887 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15e444b-9c8c-450f-a29c-b72983b55be1 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:49.240263 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15e444b-9c8c-450f-a29c-b72983b55be1 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:49.240616 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15e444b-9c8c-450f-a29c-b72983b55be1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:49.251648 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a15e444b-9c8c-450f-a29c-b72983b55be1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:49.253188 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a15e444b-9c8c-450f-a29c-b72983b55be1 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:49.262921 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55556664-d38d-4d02-98f5-36ac8a84ea5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:49.263606 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55556664-d38d-4d02-98f5-36ac8a84ea5c None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:49.263951 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55556664-d38d-4d02-98f5-36ac8a84ea5c None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:49.264291 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55556664-d38d-4d02-98f5-36ac8a84ea5c None admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:49.265007 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55556664-d38d-4d02-98f5-36ac8a84ea5c None admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:49.265377 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55556664-d38d-4d02-98f5-36ac8a84ea5c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:49.266893 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55556664-d38d-4d02-98f5-36ac8a84ea5c None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:49.268225 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-a15e444b-9c8c-450f-a29c-b72983b55be1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:49.268225 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:49.268225 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:49.268225 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:49.268225 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:49.268225 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:49.268225 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:49.268225 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:49.268225 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:49.268225 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:49.268225 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:49.268225 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:49.268225 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:39:49.268225 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:39:49.268225 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:39:49.268225 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:39:49.269155 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:49.269155 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:49.269155 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:49.269155 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:49.269155 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:49.269155 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:49.269155 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:49.269155 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:49.269155 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:49.269155 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:49.269155 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:49.269155 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:49.269155 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:49.269155 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:49.269155 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:49.269155 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:49.269155 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:39:49.270039 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:39:49.270039 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:49.270039 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:49.270039 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 329/646] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:39:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:49.270274 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a15e444b-9c8c-450f-a29c-b72983b55be1 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:49.290363 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 318/647] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:39:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:49.290363 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55556664-d38d-4d02-98f5-36ac8a84ea5c None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:49.308730 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:49.356432 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:49.386407 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a15e444b-9c8c-450f-a29c-b72983b55be1 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:49.395890 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55556664-d38d-4d02-98f5-36ac8a84ea5c None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:49.422081 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e4c9793-bf87-47d5-ba61-4b7adb93c81e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:49.422721 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4c9793-bf87-47d5-ba61-4b7adb93c81e None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:49.423029 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4c9793-bf87-47d5-ba61-4b7adb93c81e None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:49.423294 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4c9793-bf87-47d5-ba61-4b7adb93c81e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:49.434005 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71cc66af-d0f1-43ac-9ebd-2be13777a882 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:49.435022 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71cc66af-d0f1-43ac-9ebd-2be13777a882 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:49.435319 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71cc66af-d0f1-43ac-9ebd-2be13777a882 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:49.435591 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71cc66af-d0f1-43ac-9ebd-2be13777a882 None admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:49.436453 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71cc66af-d0f1-43ac-9ebd-2be13777a882 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:49.436761 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71cc66af-d0f1-43ac-9ebd-2be13777a882 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:49.437126 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e4c9793-bf87-47d5-ba61-4b7adb93c81e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:49.438265 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71cc66af-d0f1-43ac-9ebd-2be13777a882 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:49.439070 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e4c9793-bf87-47d5-ba61-4b7adb93c81e None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:49.455725 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 319/648] 10.222.0.27 () {56 vars in 1148 bytes} [Sun Sep 11 00:39:49 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:49.455725 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71cc66af-d0f1-43ac-9ebd-2be13777a882 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:49.456524 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-8e4c9793-bf87-47d5-ba61-4b7adb93c81e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:49.456524 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:49.456524 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:49.456524 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:49.456524 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:49.456524 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:49.456524 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:49.456524 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:49.456524 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:49.456524 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:49.456524 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:49.456524 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:49.456524 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:39:49.456524 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:39:49.456524 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:39:49.456524 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:39:49.457464 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:49.457464 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:49.457464 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:49.457464 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:49.457464 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:49.457464 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:49.457464 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:49.457464 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:49.457464 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:49.457464 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:49.457464 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:49.457464 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:49.457464 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:49.457464 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:49.457464 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:49.457464 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:49.457464 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:39:49.458366 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:39:49.458366 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:49.458366 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:49.458366 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 330/649] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:39:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:49.458604 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e4c9793-bf87-47d5-ba61-4b7adb93c81e None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:49.558179 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:49.565852 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71cc66af-d0f1-43ac-9ebd-2be13777a882 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:49.577294 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e4c9793-bf87-47d5-ba61-4b7adb93c81e None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:49.589280 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:49.601291 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0de6cd0-55d3-4267-b924-460ef17f693c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:49.602053 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0de6cd0-55d3-4267-b924-460ef17f693c None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:49.602307 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0de6cd0-55d3-4267-b924-460ef17f693c None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:49.602568 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0de6cd0-55d3-4267-b924-460ef17f693c None admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:49.603232 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0de6cd0-55d3-4267-b924-460ef17f693c None admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:49.603612 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0de6cd0-55d3-4267-b924-460ef17f693c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:49.604892 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0de6cd0-55d3-4267-b924-460ef17f693c None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:49.615411 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c663552c-8da3-4f02-8ccc-9d27b1fdda29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:49.616048 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c663552c-8da3-4f02-8ccc-9d27b1fdda29 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:49.616907 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c663552c-8da3-4f02-8ccc-9d27b1fdda29 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:49.617213 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c663552c-8da3-4f02-8ccc-9d27b1fdda29 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:49.623118 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 320/650] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:39:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:49.623118 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0de6cd0-55d3-4267-b924-460ef17f693c None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:49.630435 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c663552c-8da3-4f02-8ccc-9d27b1fdda29 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:49.632869 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c663552c-8da3-4f02-8ccc-9d27b1fdda29 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:49.644739 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-c663552c-8da3-4f02-8ccc-9d27b1fdda29 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 11 00:39:49.644739 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:49.644739 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:49.644739 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:49.644739 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:49.644739 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:49.644739 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:49.644739 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:49.644739 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:49.644739 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:49.644739 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:49.644739 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:49.644739 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 11 00:39:49.644739 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 11 00:39:49.644739 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 11 00:39:49.644739 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 11 00:39:49.645873 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:49.645873 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:49.645873 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 11 00:39:49.645873 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:39:49.645873 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 11 00:39:49.645873 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:39:49.645873 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:49.645873 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:49.645873 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:49.645873 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:49.645873 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:49.645873 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:49.645873 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:49.645873 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:49.645873 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:49.645873 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:49.645873 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:49.645873 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:49.647043 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:49.647043 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:49.647043 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 11 00:39:49.647043 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 11 00:39:49.647043 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 11 00:39:49.647043 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 11 00:39:49.647043 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 11 00:39:49.647043 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 11 00:39:49.647043 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 11 00:39:49.647043 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:49.650224 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 331/651] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:39:49 2022] GET /identity/v3/users/cinder => generated 84 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:49.651071 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c663552c-8da3-4f02-8ccc-9d27b1fdda29 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:49.730612 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0de6cd0-55d3-4267-b924-460ef17f693c None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:49.765240 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c663552c-8da3-4f02-8ccc-9d27b1fdda29 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:49.767412 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac856938-77ca-4930-b1cb-cb920c992ede None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:49.768017 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac856938-77ca-4930-b1cb-cb920c992ede None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:49.768288 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac856938-77ca-4930-b1cb-cb920c992ede None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:49.768554 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac856938-77ca-4930-b1cb-cb920c992ede None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:49.769270 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac856938-77ca-4930-b1cb-cb920c992ede None admin] RBAC: Adding query filter params (role.id=0390f926e7ef4715ae18697033bde90e, scope.project.id=b504a5058bc6474c8cc077dfeec16799, user.id=835def0e6c6d4365a2767051509eb0d6) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:49.769568 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac856938-77ca-4930-b1cb-cb920c992ede None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:49.771008 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac856938-77ca-4930-b1cb-cb920c992ede None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:49.778074 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 321/652] 10.222.0.27 () {56 vars in 1374 bytes} [Sun Sep 11 00:39:49 2022] GET /identity/v3/role_assignments?user.id=835def0e6c6d4365a2767051509eb0d6&scope.project.id=b504a5058bc6474c8cc077dfeec16799&role.id=0390f926e7ef4715ae18697033bde90e => generated 608 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:49.778899 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac856938-77ca-4930-b1cb-cb920c992ede None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:49.815196 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a0cfb2-5772-4c3e-8219-66cdcf8b86c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:49.816128 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a0cfb2-5772-4c3e-8219-66cdcf8b86c9 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:49.816574 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a0cfb2-5772-4c3e-8219-66cdcf8b86c9 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:49.816998 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a0cfb2-5772-4c3e-8219-66cdcf8b86c9 None admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:49.818012 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42a0cfb2-5772-4c3e-8219-66cdcf8b86c9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:49.818893 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42a0cfb2-5772-4c3e-8219-66cdcf8b86c9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:49.820519 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42a0cfb2-5772-4c3e-8219-66cdcf8b86c9 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:49.834629 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 332/653] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:39:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:49.835865 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a0cfb2-5772-4c3e-8219-66cdcf8b86c9 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:49.841415 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:49.845276 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:49.904627 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac856938-77ca-4930-b1cb-cb920c992ede None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:49.942082 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de9080c-27b3-4a0e-b396-7c3f4e0e9677 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:49.942883 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de9080c-27b3-4a0e-b396-7c3f4e0e9677 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:49.943224 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de9080c-27b3-4a0e-b396-7c3f4e0e9677 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:49.943571 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de9080c-27b3-4a0e-b396-7c3f4e0e9677 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:49.952466 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2de9080c-27b3-4a0e-b396-7c3f4e0e9677 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:49.954064 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2de9080c-27b3-4a0e-b396-7c3f4e0e9677 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:49.962811 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a0cfb2-5772-4c3e-8219-66cdcf8b86c9 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:49.966756 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-2de9080c-27b3-4a0e-b396-7c3f4e0e9677 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 11 00:39:49.966756 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:49.966756 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:49.966756 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:49.966756 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:49.966756 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:49.966756 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:49.966756 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:49.966756 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:49.966756 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:49.966756 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:49.966756 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:49.966756 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 11 00:39:49.966756 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 11 00:39:49.966756 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 11 00:39:49.966756 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 11 00:39:49.967896 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:49.967896 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:49.967896 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 11 00:39:49.967896 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:39:49.967896 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 11 00:39:49.967896 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:39:49.967896 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:49.967896 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:49.967896 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:49.967896 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:49.967896 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:49.967896 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:49.967896 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:49.967896 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:49.967896 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:49.967896 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:49.967896 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:49.967896 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:49.969054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:49.969054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:49.969054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 11 00:39:49.969054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 11 00:39:49.969054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 11 00:39:49.969054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 11 00:39:49.969054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 11 00:39:49.969054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 11 00:39:49.969054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 11 00:39:49.969054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:49.969054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 322/654] 10.222.0.27 () {56 vars in 1102 bytes} [Sun Sep 11 00:39:49 2022] GET /identity/v3/users/neutron => generated 85 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:49.970827 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1981851-5343-4ba3-bf45-8355aa8851d6 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:49.970827 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1981851-5343-4ba3-bf45-8355aa8851d6 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:49.970827 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1981851-5343-4ba3-bf45-8355aa8851d6 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:49.971917 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 323/655] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:39:49 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:39:49.976848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1981851-5343-4ba3-bf45-8355aa8851d6 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:50.008255 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ecc0bc-1376-42be-a43c-fbca751a8d03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:50.008926 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ecc0bc-1376-42be-a43c-fbca751a8d03 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:50.010075 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ecc0bc-1376-42be-a43c-fbca751a8d03 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:50.010075 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ecc0bc-1376-42be-a43c-fbca751a8d03 None admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:50.010284 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45ecc0bc-1376-42be-a43c-fbca751a8d03 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:50.010868 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45ecc0bc-1376-42be-a43c-fbca751a8d03 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:50.011978 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45ecc0bc-1376-42be-a43c-fbca751a8d03 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:50.034408 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 333/656] 10.222.0.27 () {56 vars in 1137 bytes} [Sun Sep 11 00:39:49 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:50.035803 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ecc0bc-1376-42be-a43c-fbca751a8d03 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:50.105457 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:50.112668 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1981851-5343-4ba3-bf45-8355aa8851d6 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:50.122938 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:50.149497 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ecc0bc-1376-42be-a43c-fbca751a8d03 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:50.151389 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83676312-07a1-434f-901f-334dea103f59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:50.152255 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83676312-07a1-434f-901f-334dea103f59 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:50.152628 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83676312-07a1-434f-901f-334dea103f59 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:50.153002 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83676312-07a1-434f-901f-334dea103f59 None admin] PATH_INFO: `/v3/users/glance` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:50.161501 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83676312-07a1-434f-901f-334dea103f59 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:50.163154 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83676312-07a1-434f-901f-334dea103f59 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:50.175606 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-83676312-07a1-434f-901f-334dea103f59 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 11 00:39:50.175606 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:50.175606 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:50.175606 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:50.175606 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:50.175606 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:50.175606 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:50.175606 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:50.175606 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:50.175606 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:50.175606 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:50.175606 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:50.175606 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 11 00:39:50.175606 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 11 00:39:50.175606 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 11 00:39:50.175606 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 11 00:39:50.176769 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:50.176769 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:50.176769 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 11 00:39:50.176769 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:39:50.176769 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 11 00:39:50.176769 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:39:50.176769 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:50.176769 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:50.176769 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:50.176769 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:50.176769 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:50.176769 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:50.176769 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:50.176769 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:50.176769 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:50.176769 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:50.176769 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:50.176769 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:50.177925 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:50.177925 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:50.177925 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 11 00:39:50.177925 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 11 00:39:50.177925 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 11 00:39:50.177925 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 11 00:39:50.177925 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 11 00:39:50.177925 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 11 00:39:50.177925 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 11 00:39:50.177925 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:50.179483 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 324/657] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:39:49 2022] GET /identity/v3/users/glance => generated 84 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:50.180906 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a7f0de5-57f9-4e8d-bcb5-c30da55ef518 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:50.181188 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a7f0de5-57f9-4e8d-bcb5-c30da55ef518 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:50.181460 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a7f0de5-57f9-4e8d-bcb5-c30da55ef518 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:50.192378 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d54a51-cdbf-4fc9-b94c-15f81b0e916e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:50.192993 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d54a51-cdbf-4fc9-b94c-15f81b0e916e None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:50.193270 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d54a51-cdbf-4fc9-b94c-15f81b0e916e None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:50.193540 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d54a51-cdbf-4fc9-b94c-15f81b0e916e None admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:50.194428 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d54a51-cdbf-4fc9-b94c-15f81b0e916e None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:50.194777 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d54a51-cdbf-4fc9-b94c-15f81b0e916e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:50.196239 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d54a51-cdbf-4fc9-b94c-15f81b0e916e None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:50.199743 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-6a7f0de5-57f9-4e8d-bcb5-c30da55ef518 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:39:50.200931 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-6a7f0de5-57f9-4e8d-bcb5-c30da55ef518 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:39:50.219750 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 334/658] 10.222.0.27 () {56 vars in 1139 bytes} [Sun Sep 11 00:39:50 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:50.220596 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d54a51-cdbf-4fc9-b94c-15f81b0e916e None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:50.270837 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 325/659] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:39:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:39:50.270837 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a7f0de5-57f9-4e8d-bcb5-c30da55ef518 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:50.336266 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d54a51-cdbf-4fc9-b94c-15f81b0e916e None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:50.360766 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:50.377822 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de5f4ef3-ce40-4ed5-8c37-32749f416126 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:50.378443 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5f4ef3-ce40-4ed5-8c37-32749f416126 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:50.378720 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5f4ef3-ce40-4ed5-8c37-32749f416126 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:50.379018 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5f4ef3-ce40-4ed5-8c37-32749f416126 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:50.395737 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:50.401148 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a7f0de5-57f9-4e8d-bcb5-c30da55ef518 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:50.447125 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3682f429-5fcc-4e5b-89ca-4d9a9b0910bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:50.448141 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3682f429-5fcc-4e5b-89ca-4d9a9b0910bf None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:50.448882 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3682f429-5fcc-4e5b-89ca-4d9a9b0910bf None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:50.449569 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3682f429-5fcc-4e5b-89ca-4d9a9b0910bf None admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:50.450575 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3682f429-5fcc-4e5b-89ca-4d9a9b0910bf None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:50.451078 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3682f429-5fcc-4e5b-89ca-4d9a9b0910bf None admin] RBAC: Authorizing `identity:list_users()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:50.452827 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3682f429-5fcc-4e5b-89ca-4d9a9b0910bf None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:50.455432 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de5f4ef3-ce40-4ed5-8c37-32749f416126 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:50.457304 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de5f4ef3-ce40-4ed5-8c37-32749f416126 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:50.476321 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 326/660] 10.222.0.27 () {56 vars in 1137 bytes} [Sun Sep 11 00:39:50 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:50.477394 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3682f429-5fcc-4e5b-89ca-4d9a9b0910bf None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:50.575411 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 335/661] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:39:50 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 355 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 11 00:39:50.576503 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de5f4ef3-ce40-4ed5-8c37-32749f416126 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:50.600213 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:50.604461 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3682f429-5fcc-4e5b-89ca-4d9a9b0910bf None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:50.641815 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27122e18-8513-4477-b5e9-ff7df9952224 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:50.642423 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27122e18-8513-4477-b5e9-ff7df9952224 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:50.642696 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27122e18-8513-4477-b5e9-ff7df9952224 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:50.643071 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27122e18-8513-4477-b5e9-ff7df9952224 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:50.669771 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:50.680764 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de5f4ef3-ce40-4ed5-8c37-32749f416126 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:50.715586 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27122e18-8513-4477-b5e9-ff7df9952224 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:50.716615 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c88497b3-666b-4e3f-ad0e-d0fa152e5207 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:50.717060 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27122e18-8513-4477-b5e9-ff7df9952224 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:50.717249 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88497b3-666b-4e3f-ad0e-d0fa152e5207 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:50.717491 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88497b3-666b-4e3f-ad0e-d0fa152e5207 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:50.717758 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88497b3-666b-4e3f-ad0e-d0fa152e5207 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:50.790311 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c88497b3-666b-4e3f-ad0e-d0fa152e5207 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:50.791810 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c88497b3-666b-4e3f-ad0e-d0fa152e5207 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:50.813736 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:50.822887 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 327/662] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:39:50 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 346 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 11 00:39:50.823722 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27122e18-8513-4477-b5e9-ff7df9952224 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:50.883642 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:50.907326 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 336/663] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:39:50 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 331 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 11 00:39:50.908295 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c88497b3-666b-4e3f-ad0e-d0fa152e5207 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:50.934883 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27122e18-8513-4477-b5e9-ff7df9952224 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:50.971354 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b687f3ef-8fdd-459c-bbc0-807f918dd0c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:50.971908 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b687f3ef-8fdd-459c-bbc0-807f918dd0c4 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:50.972186 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b687f3ef-8fdd-459c-bbc0-807f918dd0c4 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:50.972455 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b687f3ef-8fdd-459c-bbc0-807f918dd0c4 None admin] PATH_INFO: `/v3/roles/service` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:50.982536 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b687f3ef-8fdd-459c-bbc0-807f918dd0c4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:39:50.987396 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-b687f3ef-8fdd-459c-bbc0-807f918dd0c4 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 11 00:39:50.987396 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:50.987396 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:50.987396 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:50.987396 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:50.987396 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:50.987396 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:50.987396 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:50.987396 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:50.987396 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:50.987396 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:50.987396 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:50.987396 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:39:50.987396 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:39:50.987396 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:39:50.987396 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:39:50.988345 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:39:50.988345 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:39:50.988345 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:39:50.988345 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:39:50.988345 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:50.988345 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:50.988345 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:50.988345 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:50.988345 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:50.988345 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:50.988345 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:50.988345 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:50.988345 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:50.988345 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:50.988345 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:50.988345 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:50.989306 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:50.989306 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:50.989306 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:50.989306 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:50.989306 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:39:50.989306 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:39:50.989306 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:39:50.989306 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:39:50.989306 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:39:50.989306 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:39:50.989306 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 11 00:39:50.989306 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:50.989306 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 328/664] 10.222.0.27 () {56 vars in 1102 bytes} [Sun Sep 11 00:39:50 2022] GET /identity/v3/roles/service => generated 85 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:50.989306 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b687f3ef-8fdd-459c-bbc0-807f918dd0c4 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:51.019623 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c88497b3-666b-4e3f-ad0e-d0fa152e5207 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:51.040460 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:51.056679 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6893145-f9ea-4988-99c2-8e2dd1d157b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:51.057397 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6893145-f9ea-4988-99c2-8e2dd1d157b6 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:51.057746 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6893145-f9ea-4988-99c2-8e2dd1d157b6 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:51.058094 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6893145-f9ea-4988-99c2-8e2dd1d157b6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:51.068795 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6893145-f9ea-4988-99c2-8e2dd1d157b6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:51.070962 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6893145-f9ea-4988-99c2-8e2dd1d157b6 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:51.090012 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-a6893145-f9ea-4988-99c2-8e2dd1d157b6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:51.090012 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:51.090012 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:51.090012 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:51.090012 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:51.090012 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:51.090012 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:51.090012 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:51.090012 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:51.090012 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:51.090012 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:51.090012 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:51.090012 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:39:51.090012 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:39:51.090012 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:39:51.090012 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:39:51.091190 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:51.091190 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:51.091190 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:51.091190 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:51.091190 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:51.091190 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:51.091190 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:51.091190 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:51.091190 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:51.091190 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:51.091190 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:51.091190 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:51.091190 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:51.091190 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:51.091190 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:51.091190 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:51.091190 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:39:51.092132 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:39:51.092132 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:51.092132 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:51.092132 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 337/665] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:39:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:51.092132 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6893145-f9ea-4988-99c2-8e2dd1d157b6 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:51.109846 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b687f3ef-8fdd-459c-bbc0-807f918dd0c4 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:51.128198 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:51.146914 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8563e89e-f324-45a8-86a7-f189d212b1b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:51.147653 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8563e89e-f324-45a8-86a7-f189d212b1b4 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:51.147996 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8563e89e-f324-45a8-86a7-f189d212b1b4 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:51.148333 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8563e89e-f324-45a8-86a7-f189d212b1b4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:51.158489 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8563e89e-f324-45a8-86a7-f189d212b1b4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:51.160056 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8563e89e-f324-45a8-86a7-f189d212b1b4 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:51.176431 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-8563e89e-f324-45a8-86a7-f189d212b1b4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:51.176431 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:51.176431 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:51.176431 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:51.176431 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:51.176431 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:51.176431 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:51.176431 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:51.176431 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:51.176431 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:51.176431 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:51.176431 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:51.176431 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:39:51.176431 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:39:51.176431 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:39:51.176431 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:39:51.177602 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:51.177602 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:51.177602 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:51.177602 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:51.177602 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:51.177602 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:51.177602 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:51.177602 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:51.177602 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:51.177602 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:51.177602 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:51.177602 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:51.177602 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:51.177602 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:51.177602 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:51.177602 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:51.177602 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:39:51.178574 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:39:51.178574 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:51.178574 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:51.178574 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 329/666] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:39:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:51.179685 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8563e89e-f324-45a8-86a7-f189d212b1b4 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:51.204957 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6893145-f9ea-4988-99c2-8e2dd1d157b6 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:51.240723 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-906943df-8490-4901-a647-f94190913973 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:51.241923 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906943df-8490-4901-a647-f94190913973 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:51.242201 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906943df-8490-4901-a647-f94190913973 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:51.242466 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906943df-8490-4901-a647-f94190913973 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:51.253247 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-906943df-8490-4901-a647-f94190913973 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:51.254700 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-906943df-8490-4901-a647-f94190913973 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:51.269398 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-906943df-8490-4901-a647-f94190913973 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:51.269398 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:51.269398 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:51.269398 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:51.269398 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:51.269398 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:51.269398 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:51.269398 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:51.269398 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:51.269398 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:51.269398 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:51.269398 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:51.269398 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:39:51.269398 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:39:51.269398 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:39:51.269398 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:39:51.270351 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:51.270351 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:51.270351 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:51.270351 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:51.270351 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:51.270351 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:51.270351 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:51.270351 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:51.270351 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:51.270351 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:51.270351 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:51.270351 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:51.270351 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:51.270351 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:51.270351 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:51.270351 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:51.270351 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:39:51.271637 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:39:51.271637 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:51.271637 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:51.271637 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 338/667] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:39:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:51.271637 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-906943df-8490-4901-a647-f94190913973 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:51.282359 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8563e89e-f324-45a8-86a7-f189d212b1b4 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:51.302554 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:51.321629 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee7cb079-31bc-495d-8b89-7f10a9f39fd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:51.322309 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7cb079-31bc-495d-8b89-7f10a9f39fd2 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:51.322649 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7cb079-31bc-495d-8b89-7f10a9f39fd2 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:51.323336 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7cb079-31bc-495d-8b89-7f10a9f39fd2 None admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:51.324062 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee7cb079-31bc-495d-8b89-7f10a9f39fd2 None admin] RBAC: Adding query filter params (name=service) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:51.324711 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee7cb079-31bc-495d-8b89-7f10a9f39fd2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:51.326154 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee7cb079-31bc-495d-8b89-7f10a9f39fd2 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:51.338827 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 330/668] 10.222.0.27 () {56 vars in 1103 bytes} [Sun Sep 11 00:39:51 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:51.338827 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee7cb079-31bc-495d-8b89-7f10a9f39fd2 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:51.371543 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:51.386837 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-906943df-8490-4901-a647-f94190913973 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:51.422683 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a07cfb-0b70-4c80-98b9-c79b912be293 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:51.423339 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a07cfb-0b70-4c80-98b9-c79b912be293 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:51.423605 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a07cfb-0b70-4c80-98b9-c79b912be293 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:51.423874 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a07cfb-0b70-4c80-98b9-c79b912be293 None admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:51.424627 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34a07cfb-0b70-4c80-98b9-c79b912be293 None admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:51.424812 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34a07cfb-0b70-4c80-98b9-c79b912be293 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:51.426178 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34a07cfb-0b70-4c80-98b9-c79b912be293 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:51.440249 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 339/669] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:39:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:51.441337 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a07cfb-0b70-4c80-98b9-c79b912be293 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:51.451540 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee7cb079-31bc-495d-8b89-7f10a9f39fd2 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:51.490077 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9234c91c-9539-40c0-a1d6-9db3869c4e9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:51.491453 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9234c91c-9539-40c0-a1d6-9db3869c4e9d None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:51.491903 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9234c91c-9539-40c0-a1d6-9db3869c4e9d None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:51.492232 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9234c91c-9539-40c0-a1d6-9db3869c4e9d None admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:51.492816 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9234c91c-9539-40c0-a1d6-9db3869c4e9d None admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:51.493108 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9234c91c-9539-40c0-a1d6-9db3869c4e9d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:51.494480 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9234c91c-9539-40c0-a1d6-9db3869c4e9d None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:51.511773 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 331/670] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:39:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:51.511773 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9234c91c-9539-40c0-a1d6-9db3869c4e9d None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:51.538126 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:51.556322 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a07cfb-0b70-4c80-98b9-c79b912be293 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:51.591314 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c738f82-8d4e-4d88-8609-684a62b05199 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:51.591920 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c738f82-8d4e-4d88-8609-684a62b05199 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:51.592199 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c738f82-8d4e-4d88-8609-684a62b05199 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:51.592469 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c738f82-8d4e-4d88-8609-684a62b05199 None admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:51.593116 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c738f82-8d4e-4d88-8609-684a62b05199 None admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:51.593416 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c738f82-8d4e-4d88-8609-684a62b05199 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:51.594777 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c738f82-8d4e-4d88-8609-684a62b05199 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:51.606835 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 340/671] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:39:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:51.607430 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c738f82-8d4e-4d88-8609-684a62b05199 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:51.613202 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:51.625346 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9234c91c-9539-40c0-a1d6-9db3869c4e9d None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:51.660528 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcdfc3f3-afa5-494b-8cc5-411e172b3932 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:51.661156 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcdfc3f3-afa5-494b-8cc5-411e172b3932 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:51.661433 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcdfc3f3-afa5-494b-8cc5-411e172b3932 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:51.661701 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcdfc3f3-afa5-494b-8cc5-411e172b3932 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:51.671968 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcdfc3f3-afa5-494b-8cc5-411e172b3932 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:51.673422 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcdfc3f3-afa5-494b-8cc5-411e172b3932 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:51.689011 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-fcdfc3f3-afa5-494b-8cc5-411e172b3932 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:51.689011 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:51.689011 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:51.689011 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:51.689011 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:51.689011 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:51.689011 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:51.689011 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:51.689011 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:51.689011 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:51.689011 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:51.689011 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:51.689011 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:39:51.689011 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:39:51.689011 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:39:51.689011 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:39:51.690040 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:51.690040 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:51.690040 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:51.690040 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:51.690040 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:51.690040 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:51.690040 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:51.690040 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:51.690040 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:51.690040 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:51.690040 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:51.690040 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:51.690040 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:51.690040 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:51.690040 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:51.690040 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:51.690040 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:39:51.695968 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:39:51.695968 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:51.695968 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:51.695968 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 332/672] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:39:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:51.695968 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcdfc3f3-afa5-494b-8cc5-411e172b3932 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:51.722301 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c738f82-8d4e-4d88-8609-684a62b05199 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:51.758246 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6adaa3fc-19a9-4e30-8d0a-75a8180f1b71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:51.759146 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6adaa3fc-19a9-4e30-8d0a-75a8180f1b71 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:51.759579 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6adaa3fc-19a9-4e30-8d0a-75a8180f1b71 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:51.759854 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6adaa3fc-19a9-4e30-8d0a-75a8180f1b71 None admin] PATH_INFO: `/v3/projects/service` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:51.767136 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6adaa3fc-19a9-4e30-8d0a-75a8180f1b71 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:51.768597 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6adaa3fc-19a9-4e30-8d0a-75a8180f1b71 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:51.781229 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-6adaa3fc-19a9-4e30-8d0a-75a8180f1b71 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:39:51.781229 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:51.781229 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:51.781229 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:51.781229 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:51.781229 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:51.781229 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:51.781229 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:51.781229 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:51.781229 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:51.781229 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:51.781229 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:51.781229 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 11 00:39:51.781229 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 11 00:39:51.781229 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 11 00:39:51.782192 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 11 00:39:51.782192 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:51.782192 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:51.782192 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:51.782192 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:51.782192 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:51.782192 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:51.782192 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:51.782192 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:51.782192 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:51.782192 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:51.782192 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:51.782192 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:51.782192 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:51.782192 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:51.782192 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:51.782192 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:51.783093 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 11 00:39:51.783093 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 11 00:39:51.783093 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 11 00:39:51.783093 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 11 00:39:51.783093 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 11 00:39:51.783093 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 11 00:39:51.783093 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:39:51.783093 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:51.783093 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 341/673] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:39:51 2022] GET /identity/v3/projects/service => generated 88 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:51.783591 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6adaa3fc-19a9-4e30-8d0a-75a8180f1b71 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:51.787749 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:51.807866 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcdfc3f3-afa5-494b-8cc5-411e172b3932 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:51.842043 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f36a2f5-425e-43cf-a87b-2b4420d21250 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:51.842664 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f36a2f5-425e-43cf-a87b-2b4420d21250 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:51.842966 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f36a2f5-425e-43cf-a87b-2b4420d21250 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:51.843242 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f36a2f5-425e-43cf-a87b-2b4420d21250 None admin] PATH_INFO: `/v3/projects/service` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:51.850780 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f36a2f5-425e-43cf-a87b-2b4420d21250 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:51.853115 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f36a2f5-425e-43cf-a87b-2b4420d21250 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:51.864598 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-0f36a2f5-425e-43cf-a87b-2b4420d21250 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:39:51.864598 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:51.864598 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:51.864598 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:51.864598 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:51.864598 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:51.864598 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:51.864598 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:51.864598 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:51.864598 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:51.864598 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:51.864598 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:51.864598 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 11 00:39:51.864598 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 11 00:39:51.864598 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 11 00:39:51.865523 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 11 00:39:51.865523 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:51.865523 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:51.865523 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:51.865523 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:51.865523 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:51.865523 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:51.865523 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:51.865523 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:51.865523 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:51.865523 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:51.865523 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:51.865523 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:51.865523 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:51.865523 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:51.865523 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:51.865523 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:51.866486 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 11 00:39:51.866486 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 11 00:39:51.866486 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 11 00:39:51.866486 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 11 00:39:51.866486 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 11 00:39:51.866486 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 11 00:39:51.866486 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:39:51.866486 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:51.870693 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 333/674] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:39:51 2022] GET /identity/v3/projects/service => generated 88 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:51.871556 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f36a2f5-425e-43cf-a87b-2b4420d21250 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:51.874290 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:51.891625 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6adaa3fc-19a9-4e30-8d0a-75a8180f1b71 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:51.927644 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b81e0f3-524a-413c-b403-e6ce04bdd03c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:51.928282 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b81e0f3-524a-413c-b403-e6ce04bdd03c None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:51.928618 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b81e0f3-524a-413c-b403-e6ce04bdd03c None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:51.928915 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b81e0f3-524a-413c-b403-e6ce04bdd03c None admin] PATH_INFO: `/v3/projects/service` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:51.936608 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b81e0f3-524a-413c-b403-e6ce04bdd03c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:51.938090 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b81e0f3-524a-413c-b403-e6ce04bdd03c None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:51.952544 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-4b81e0f3-524a-413c-b403-e6ce04bdd03c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:39:51.952544 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:51.952544 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:51.952544 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:51.952544 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:51.952544 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:51.952544 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:51.952544 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:51.952544 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:51.952544 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:51.952544 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:51.952544 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:51.952544 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 11 00:39:51.952544 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 11 00:39:51.952544 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 11 00:39:51.953709 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 11 00:39:51.953709 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:51.953709 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:51.953709 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:51.953709 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:51.953709 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:51.953709 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:51.953709 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:51.953709 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:51.953709 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:51.953709 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:51.953709 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:51.953709 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:51.953709 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:51.953709 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:51.953709 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:51.953709 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:51.954646 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 11 00:39:51.954646 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 11 00:39:51.954646 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 11 00:39:51.954646 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 11 00:39:51.954646 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 11 00:39:51.954646 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 11 00:39:51.954646 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:39:51.954646 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:51.954646 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 342/675] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:39:51 2022] GET /identity/v3/projects/service => generated 88 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:51.955210 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b81e0f3-524a-413c-b403-e6ce04bdd03c None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:51.995289 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f36a2f5-425e-43cf-a87b-2b4420d21250 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:52.027834 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:52.030490 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c32b188-040b-408f-8149-db6e359d590c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:52.031166 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c32b188-040b-408f-8149-db6e359d590c None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:52.031457 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c32b188-040b-408f-8149-db6e359d590c None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:52.031880 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c32b188-040b-408f-8149-db6e359d590c None admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:52.032519 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c32b188-040b-408f-8149-db6e359d590c None admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:52.032831 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c32b188-040b-408f-8149-db6e359d590c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:52.034742 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c32b188-040b-408f-8149-db6e359d590c None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:52.048352 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 334/676] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:39:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:52.053375 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c32b188-040b-408f-8149-db6e359d590c None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:52.070372 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b81e0f3-524a-413c-b403-e6ce04bdd03c None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:52.117295 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e20b08-8a4d-47b5-8c3f-9ff1be17310e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:52.117960 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e20b08-8a4d-47b5-8c3f-9ff1be17310e None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:52.118233 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e20b08-8a4d-47b5-8c3f-9ff1be17310e None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:52.118499 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e20b08-8a4d-47b5-8c3f-9ff1be17310e None admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:52.119169 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:52.119340 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57e20b08-8a4d-47b5-8c3f-9ff1be17310e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:52.119515 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57e20b08-8a4d-47b5-8c3f-9ff1be17310e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:52.120905 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57e20b08-8a4d-47b5-8c3f-9ff1be17310e None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:52.132942 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 343/677] 10.222.0.27 () {56 vars in 1148 bytes} [Sun Sep 11 00:39:51 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:52.133728 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e20b08-8a4d-47b5-8c3f-9ff1be17310e None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:52.164347 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c32b188-040b-408f-8149-db6e359d590c None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:52.197963 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11f4cd94-acbe-435a-94f6-8bc61a253958 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:52.198649 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11f4cd94-acbe-435a-94f6-8bc61a253958 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:52.198961 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11f4cd94-acbe-435a-94f6-8bc61a253958 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:52.199235 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11f4cd94-acbe-435a-94f6-8bc61a253958 None admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:52.199923 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11f4cd94-acbe-435a-94f6-8bc61a253958 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:52.200227 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11f4cd94-acbe-435a-94f6-8bc61a253958 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:52.201652 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11f4cd94-acbe-435a-94f6-8bc61a253958 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:52.213070 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 335/678] 10.222.0.27 () {56 vars in 1148 bytes} [Sun Sep 11 00:39:52 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:52.213889 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11f4cd94-acbe-435a-94f6-8bc61a253958 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:52.231737 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e20b08-8a4d-47b5-8c3f-9ff1be17310e None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:52.254734 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:52.266184 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2a5606-b966-4d46-b208-da7348a2f590 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:52.266782 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2a5606-b966-4d46-b208-da7348a2f590 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:52.267087 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2a5606-b966-4d46-b208-da7348a2f590 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:52.267353 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2a5606-b966-4d46-b208-da7348a2f590 None admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:52.268032 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b2a5606-b966-4d46-b208-da7348a2f590 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:52.268330 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b2a5606-b966-4d46-b208-da7348a2f590 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:52.269839 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b2a5606-b966-4d46-b208-da7348a2f590 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:52.281200 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 344/679] 10.222.0.27 () {56 vars in 1148 bytes} [Sun Sep 11 00:39:52 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:52.282281 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b79a63b4-f92f-4920-865c-bdc871217c92 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:52.282558 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b79a63b4-f92f-4920-865c-bdc871217c92 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:52.282848 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b79a63b4-f92f-4920-865c-bdc871217c92 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:52.284025 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 345/680] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:39:52 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:39:52.284699 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b79a63b4-f92f-4920-865c-bdc871217c92 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:52.317029 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11f4cd94-acbe-435a-94f6-8bc61a253958 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:52.339272 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:52.350599 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c705dbf-ea23-4953-9da5-c37591bfc8f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:52.351480 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c705dbf-ea23-4953-9da5-c37591bfc8f0 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:52.351766 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c705dbf-ea23-4953-9da5-c37591bfc8f0 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:52.352028 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c705dbf-ea23-4953-9da5-c37591bfc8f0 None admin] PATH_INFO: `/v3/users/swift` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:52.359386 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c705dbf-ea23-4953-9da5-c37591bfc8f0 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:52.360816 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c705dbf-ea23-4953-9da5-c37591bfc8f0 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:52.372573 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-7c705dbf-ea23-4953-9da5-c37591bfc8f0 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 11 00:39:52.372573 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:52.372573 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:52.372573 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:52.372573 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:52.372573 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:52.372573 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:52.372573 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:52.372573 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:52.372573 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:52.372573 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:52.372573 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:52.372573 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 11 00:39:52.372573 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 11 00:39:52.372573 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 11 00:39:52.372573 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 11 00:39:52.373685 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:52.373685 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:52.373685 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 11 00:39:52.373685 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:39:52.373685 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 11 00:39:52.373685 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:39:52.373685 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:52.373685 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:52.373685 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:52.373685 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:52.373685 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:52.373685 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:52.373685 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:52.373685 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:52.373685 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:52.373685 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:52.373685 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:52.373685 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:52.374962 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:52.374962 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:52.374962 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 11 00:39:52.374962 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 11 00:39:52.374962 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 11 00:39:52.374962 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 11 00:39:52.374962 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 11 00:39:52.374962 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 11 00:39:52.374962 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 11 00:39:52.374962 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:52.374962 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 336/681] 10.222.0.27 () {56 vars in 1096 bytes} [Sun Sep 11 00:39:52 2022] GET /identity/v3/users/swift => generated 83 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:52.374962 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c705dbf-ea23-4953-9da5-c37591bfc8f0 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:52.391937 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b79a63b4-f92f-4920-865c-bdc871217c92 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:52.425671 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af49371c-abb0-4160-8718-9d5766fda2bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:52.426273 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af49371c-abb0-4160-8718-9d5766fda2bc None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:52.426553 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af49371c-abb0-4160-8718-9d5766fda2bc None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:52.426859 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af49371c-abb0-4160-8718-9d5766fda2bc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:52.427578 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af49371c-abb0-4160-8718-9d5766fda2bc None admin] RBAC: Adding query filter params (role.id=0390f926e7ef4715ae18697033bde90e, scope.project.id=b504a5058bc6474c8cc077dfeec16799, user.id=22374eaff0834245b88506dcd7b24ede) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:52.427888 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af49371c-abb0-4160-8718-9d5766fda2bc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:52.429968 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af49371c-abb0-4160-8718-9d5766fda2bc None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:52.436001 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 346/682] 10.222.0.27 () {56 vars in 1374 bytes} [Sun Sep 11 00:39:52 2022] GET /identity/v3/role_assignments?user.id=22374eaff0834245b88506dcd7b24ede&scope.project.id=b504a5058bc6474c8cc077dfeec16799&role.id=0390f926e7ef4715ae18697033bde90e => generated 608 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:52.436775 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af49371c-abb0-4160-8718-9d5766fda2bc None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:52.483925 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c705dbf-ea23-4953-9da5-c37591bfc8f0 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:52.492971 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:52.524972 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d63b951f-16bc-471e-b7a8-cf1561a1c553 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:52.525725 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63b951f-16bc-471e-b7a8-cf1561a1c553 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:52.526005 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63b951f-16bc-471e-b7a8-cf1561a1c553 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:52.526284 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63b951f-16bc-471e-b7a8-cf1561a1c553 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:52.527057 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d63b951f-16bc-471e-b7a8-cf1561a1c553 None admin] RBAC: Adding query filter params (role.id=0390f926e7ef4715ae18697033bde90e, scope.project.id=b504a5058bc6474c8cc077dfeec16799, user.id=a5f7b13283cb4ebfb25160aa6ad0e2f4) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:52.527372 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d63b951f-16bc-471e-b7a8-cf1561a1c553 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:52.528958 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d63b951f-16bc-471e-b7a8-cf1561a1c553 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:52.544858 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 337/683] 10.222.0.27 () {56 vars in 1374 bytes} [Sun Sep 11 00:39:52 2022] GET /identity/v3/role_assignments?user.id=a5f7b13283cb4ebfb25160aa6ad0e2f4&scope.project.id=b504a5058bc6474c8cc077dfeec16799&role.id=0390f926e7ef4715ae18697033bde90e => generated 608 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:52.544858 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c27d28-75e4-4786-a500-b50b8902b550 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:52.544858 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c27d28-75e4-4786-a500-b50b8902b550 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:52.544858 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c27d28-75e4-4786-a500-b50b8902b550 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:52.559768 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-b9c27d28-75e4-4786-a500-b50b8902b550 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:39:52.561038 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-b9c27d28-75e4-4786-a500-b50b8902b550 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:39:52.568049 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af49371c-abb0-4160-8718-9d5766fda2bc None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:52.613737 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:52.629588 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5970b7ca-43e4-4fc6-855e-5a916a05a07e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:52.630140 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5970b7ca-43e4-4fc6-855e-5a916a05a07e None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:52.630140 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5970b7ca-43e4-4fc6-855e-5a916a05a07e None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:52.630311 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5970b7ca-43e4-4fc6-855e-5a916a05a07e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:52.631213 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5970b7ca-43e4-4fc6-855e-5a916a05a07e None admin] RBAC: Adding query filter params (role.id=0390f926e7ef4715ae18697033bde90e, scope.project.id=b504a5058bc6474c8cc077dfeec16799, user.id=06b1c911731f4267a5f37e91d948e7e9) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:52.631342 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5970b7ca-43e4-4fc6-855e-5a916a05a07e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:52.633863 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5970b7ca-43e4-4fc6-855e-5a916a05a07e None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:52.640986 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 347/684] 10.222.0.27 () {56 vars in 1374 bytes} [Sun Sep 11 00:39:52 2022] GET /identity/v3/role_assignments?user.id=06b1c911731f4267a5f37e91d948e7e9&scope.project.id=b504a5058bc6474c8cc077dfeec16799&role.id=0390f926e7ef4715ae18697033bde90e => generated 608 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:52.641857 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5970b7ca-43e4-4fc6-855e-5a916a05a07e None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:52.673539 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 338/685] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:39:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:39:52.784783 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5970b7ca-43e4-4fc6-855e-5a916a05a07e None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:52.801958 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:52.829128 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2101ed5-d142-4274-9a27-5a9233f2a2df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:52.829742 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2101ed5-d142-4274-9a27-5a9233f2a2df None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:52.830017 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2101ed5-d142-4274-9a27-5a9233f2a2df None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:52.830281 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2101ed5-d142-4274-9a27-5a9233f2a2df None admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:52.831044 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2101ed5-d142-4274-9a27-5a9233f2a2df None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:52.831342 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2101ed5-d142-4274-9a27-5a9233f2a2df None admin] RBAC: Authorizing `identity:list_users()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:52.832757 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2101ed5-d142-4274-9a27-5a9233f2a2df None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:52.852437 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 348/686] 10.222.0.27 () {56 vars in 1135 bytes} [Sun Sep 11 00:39:52 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:52.857942 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c27d28-75e4-4786-a500-b50b8902b550 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:52.969752 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c27d28-75e4-4786-a500-b50b8902b550 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:52.973002 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2101ed5-d142-4274-9a27-5a9233f2a2df None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:53.026575 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f59991b-d755-4387-9ada-70b3cd9bb8df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:53.027262 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f59991b-d755-4387-9ada-70b3cd9bb8df None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:53.027537 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f59991b-d755-4387-9ada-70b3cd9bb8df None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:53.027805 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f59991b-d755-4387-9ada-70b3cd9bb8df None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:53.120620 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:53.134437 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2101ed5-d142-4274-9a27-5a9233f2a2df None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:53.148655 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f59991b-d755-4387-9ada-70b3cd9bb8df None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:53.150967 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f59991b-d755-4387-9ada-70b3cd9bb8df None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:53.212485 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbac7e0e-19b7-4943-8bf8-22cea6788bc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:53.213143 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbac7e0e-19b7-4943-8bf8-22cea6788bc9 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:53.213429 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbac7e0e-19b7-4943-8bf8-22cea6788bc9 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:53.213715 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbac7e0e-19b7-4943-8bf8-22cea6788bc9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:53.240196 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbac7e0e-19b7-4943-8bf8-22cea6788bc9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:39:53.246224 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-bbac7e0e-19b7-4943-8bf8-22cea6788bc9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:39:53.246224 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:53.246224 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:53.246224 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:53.246224 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:53.246224 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:53.246224 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:53.246224 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:53.246224 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:53.246224 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:53.246224 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:53.246224 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:53.246224 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:39:53.246224 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:39:53.246224 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:39:53.246224 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:39:53.247212 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:39:53.247212 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:39:53.247212 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:39:53.247212 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:39:53.247212 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:53.247212 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:53.247212 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:53.247212 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:53.247212 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:53.247212 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:53.247212 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:53.247212 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:53.247212 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:53.247212 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:53.247212 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:53.247212 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:53.248063 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:53.248063 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:53.248063 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:53.248063 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:53.248063 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:39:53.248063 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:39:53.248063 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:39:53.248063 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:39:53.248063 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:39:53.248063 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:39:53.248063 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:39:53.248063 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:53.255082 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 349/687] 10.222.0.27 () {56 vars in 1096 bytes} [Sun Sep 11 00:39:52 2022] GET /identity/v3/roles/admin => generated 83 bytes in 282 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:39:53.258030 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbac7e0e-19b7-4943-8bf8-22cea6788bc9 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:53.269844 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 339/688] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:39:52 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 412 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 11 00:39:53.275777 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f59991b-d755-4387-9ada-70b3cd9bb8df None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:53.296434 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:53.385567 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:53.425921 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f59991b-d755-4387-9ada-70b3cd9bb8df None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:53.433349 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbac7e0e-19b7-4943-8bf8-22cea6788bc9 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:53.498753 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a1317c7-33f5-469e-bd55-aa69f9d730d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:53.500245 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1317c7-33f5-469e-bd55-aa69f9d730d1 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:53.501591 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1317c7-33f5-469e-bd55-aa69f9d730d1 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:53.502581 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1317c7-33f5-469e-bd55-aa69f9d730d1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:53.509394 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c620238c-85f4-4acc-9ec0-702113edfa27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:53.514068 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c620238c-85f4-4acc-9ec0-702113edfa27 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:53.514591 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c620238c-85f4-4acc-9ec0-702113edfa27 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:53.515085 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c620238c-85f4-4acc-9ec0-702113edfa27 None admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:53.515981 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c620238c-85f4-4acc-9ec0-702113edfa27 None admin] RBAC: Adding query filter params (name=admin) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:53.516509 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c620238c-85f4-4acc-9ec0-702113edfa27 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:53.518243 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c620238c-85f4-4acc-9ec0-702113edfa27 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:53.524933 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a1317c7-33f5-469e-bd55-aa69f9d730d1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:53.529002 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a1317c7-33f5-469e-bd55-aa69f9d730d1 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:53.536536 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 350/689] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:39:53 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:53.542761 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c620238c-85f4-4acc-9ec0-702113edfa27 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:53.555957 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-8a1317c7-33f5-469e-bd55-aa69f9d730d1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:53.555957 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:53.555957 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:53.555957 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:53.555957 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:53.555957 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:53.555957 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:53.555957 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:53.555957 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:53.555957 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:53.555957 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:53.555957 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:53.555957 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:39:53.555957 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:39:53.555957 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:39:53.555957 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:39:53.556981 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:53.556981 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:53.556981 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:53.556981 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:53.556981 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:53.556981 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:53.556981 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:53.556981 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:53.556981 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:53.556981 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:53.556981 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:53.556981 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:53.556981 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:53.556981 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:53.556981 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:53.556981 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:53.556981 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:39:53.557902 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:39:53.557902 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:53.557902 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:53.557902 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 340/690] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:39:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 282 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:53.563060 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a1317c7-33f5-469e-bd55-aa69f9d730d1 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:53.646532 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:53.658341 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c620238c-85f4-4acc-9ec0-702113edfa27 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:53.698386 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f78db91a-7e0e-4d59-a00f-d720a0895d05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:53.699484 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78db91a-7e0e-4d59-a00f-d720a0895d05 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:53.700005 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78db91a-7e0e-4d59-a00f-d720a0895d05 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:53.700478 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78db91a-7e0e-4d59-a00f-d720a0895d05 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:53.714080 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f78db91a-7e0e-4d59-a00f-d720a0895d05 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:53.715979 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f78db91a-7e0e-4d59-a00f-d720a0895d05 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:53.731214 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-f78db91a-7e0e-4d59-a00f-d720a0895d05 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:53.731214 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:53.731214 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:53.731214 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:53.731214 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:53.731214 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:53.731214 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:53.731214 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:53.731214 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:53.731214 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:53.731214 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:53.731214 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:53.731214 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:39:53.731214 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:39:53.731214 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:39:53.731214 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:39:53.732194 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:53.732194 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:53.732194 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:53.732194 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:53.732194 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:53.732194 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:53.732194 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:53.732194 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:53.732194 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:53.732194 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:53.732194 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:53.732194 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:53.732194 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:53.732194 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:53.732194 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:53.732194 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:53.732194 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:39:53.733170 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:39:53.733170 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:53.733170 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:53.735296 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 351/691] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:39:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:53.747790 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f78db91a-7e0e-4d59-a00f-d720a0895d05 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:53.763259 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a1317c7-33f5-469e-bd55-aa69f9d730d1 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:53.809278 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:53.818074 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beee4596-df4e-46b6-a16e-f5794bf3b081 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:53.818671 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beee4596-df4e-46b6-a16e-f5794bf3b081 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:53.818977 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beee4596-df4e-46b6-a16e-f5794bf3b081 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:53.819275 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beee4596-df4e-46b6-a16e-f5794bf3b081 None admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:53.820514 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beee4596-df4e-46b6-a16e-f5794bf3b081 None admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:53.820817 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beee4596-df4e-46b6-a16e-f5794bf3b081 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:53.822363 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beee4596-df4e-46b6-a16e-f5794bf3b081 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:53.837243 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 341/692] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:39:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:53.842804 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beee4596-df4e-46b6-a16e-f5794bf3b081 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:53.871467 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f78db91a-7e0e-4d59-a00f-d720a0895d05 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:53.914106 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c201008-402a-4c6a-af49-b6b013e4b109 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:53.914645 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c201008-402a-4c6a-af49-b6b013e4b109 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:53.914951 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c201008-402a-4c6a-af49-b6b013e4b109 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:53.915223 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c201008-402a-4c6a-af49-b6b013e4b109 None admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:53.918809 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c201008-402a-4c6a-af49-b6b013e4b109 None admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:53.918809 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c201008-402a-4c6a-af49-b6b013e4b109 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:53.918809 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c201008-402a-4c6a-af49-b6b013e4b109 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:53.932077 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 352/693] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:39:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:53.937094 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c201008-402a-4c6a-af49-b6b013e4b109 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:53.940762 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:54.043876 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c201008-402a-4c6a-af49-b6b013e4b109 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:54.083193 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc96413-cdb7-426f-ad40-bfb5e5c2053e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:54.083839 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc96413-cdb7-426f-ad40-bfb5e5c2053e None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:54.084113 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc96413-cdb7-426f-ad40-bfb5e5c2053e None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:54.084377 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc96413-cdb7-426f-ad40-bfb5e5c2053e None admin] PATH_INFO: `/v3/users/nova` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:54.092697 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fc96413-cdb7-426f-ad40-bfb5e5c2053e None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:54.094133 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fc96413-cdb7-426f-ad40-bfb5e5c2053e None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:54.109410 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-8fc96413-cdb7-426f-ad40-bfb5e5c2053e None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 11 00:39:54.109410 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:54.109410 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:54.109410 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:54.109410 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:54.109410 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:54.109410 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:54.109410 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:54.109410 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:54.109410 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:54.109410 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:54.109410 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:54.109410 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 11 00:39:54.109410 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 11 00:39:54.109410 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 11 00:39:54.109410 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 11 00:39:54.110354 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:54.110354 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:54.110354 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 11 00:39:54.110354 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:39:54.110354 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 11 00:39:54.110354 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:39:54.110354 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:54.110354 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:54.110354 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:54.110354 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:54.110354 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:54.110354 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:54.110354 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:54.110354 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:54.110354 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:54.110354 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:54.110354 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:54.110354 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:54.111917 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:54.111917 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:54.111917 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 11 00:39:54.111917 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 11 00:39:54.111917 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 11 00:39:54.111917 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 11 00:39:54.111917 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 11 00:39:54.111917 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 11 00:39:54.111917 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 11 00:39:54.111917 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:54.115427 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 353/694] 10.222.0.27 () {56 vars in 1093 bytes} [Sun Sep 11 00:39:53 2022] GET /identity/v3/users/nova => generated 82 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:54.117449 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc96413-cdb7-426f-ad40-bfb5e5c2053e None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:54.142825 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beee4596-df4e-46b6-a16e-f5794bf3b081 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:54.182826 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:54.226230 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc96413-cdb7-426f-ad40-bfb5e5c2053e None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:54.231293 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-524f8a9a-471f-4f9b-baa4-4160240f51f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:54.231940 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-524f8a9a-471f-4f9b-baa4-4160240f51f0 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:54.232220 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-524f8a9a-471f-4f9b-baa4-4160240f51f0 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:54.232490 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-524f8a9a-471f-4f9b-baa4-4160240f51f0 None admin] PATH_INFO: `/v3/projects/service` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:54.240714 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-524f8a9a-471f-4f9b-baa4-4160240f51f0 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:54.242548 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-524f8a9a-471f-4f9b-baa4-4160240f51f0 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:54.263520 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d720a7b-4086-4d31-963d-5b01e97b1cc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:54.264282 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d720a7b-4086-4d31-963d-5b01e97b1cc3 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:54.264645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-524f8a9a-471f-4f9b-baa4-4160240f51f0 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:39:54.264645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:54.264645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:54.264645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:54.264645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:54.264645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:54.264645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:54.264645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:54.264645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:54.264645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:54.264645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:54.264645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:54.264645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 11 00:39:54.264645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 11 00:39:54.264645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 11 00:39:54.265533 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 11 00:39:54.265533 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:54.265533 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:54.265533 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:54.265533 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:54.265533 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:54.265533 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:54.265533 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:54.265533 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:54.265533 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:54.265533 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:54.265533 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:54.265533 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:54.265533 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:54.265533 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:54.265533 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:54.265533 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:54.266608 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 11 00:39:54.266608 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 11 00:39:54.266608 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 11 00:39:54.266608 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 11 00:39:54.266608 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 11 00:39:54.266608 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 11 00:39:54.266608 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:39:54.266608 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:54.267461 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d720a7b-4086-4d31-963d-5b01e97b1cc3 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:54.267461 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d720a7b-4086-4d31-963d-5b01e97b1cc3 None admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:54.267461 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d720a7b-4086-4d31-963d-5b01e97b1cc3 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:54.267461 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d720a7b-4086-4d31-963d-5b01e97b1cc3 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:54.267728 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 342/695] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:39:53 2022] GET /identity/v3/projects/service => generated 88 bytes in 423 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:39:54.267798 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d720a7b-4086-4d31-963d-5b01e97b1cc3 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:54.271236 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-524f8a9a-471f-4f9b-baa4-4160240f51f0 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:54.300363 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 354/696] 10.222.0.27 () {56 vars in 1133 bytes} [Sun Sep 11 00:39:54 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:54.307513 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d720a7b-4086-4d31-963d-5b01e97b1cc3 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:54.370962 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:54.438031 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-524f8a9a-471f-4f9b-baa4-4160240f51f0 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:54.477797 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a70113-3e63-4f96-9e0b-f74020de6f18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:54.478405 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a70113-3e63-4f96-9e0b-f74020de6f18 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:54.478722 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a70113-3e63-4f96-9e0b-f74020de6f18 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:54.479026 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a70113-3e63-4f96-9e0b-f74020de6f18 None admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:54.479707 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5a70113-3e63-4f96-9e0b-f74020de6f18 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:54.480005 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5a70113-3e63-4f96-9e0b-f74020de6f18 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:54.481449 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5a70113-3e63-4f96-9e0b-f74020de6f18 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:54.496543 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 343/697] 10.222.0.27 () {56 vars in 1148 bytes} [Sun Sep 11 00:39:54 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:39:54.502434 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a70113-3e63-4f96-9e0b-f74020de6f18 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:54.509534 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d720a7b-4086-4d31-963d-5b01e97b1cc3 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:54.559705 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:54.586585 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c32cac67-61bf-4036-a203-e6cd3e946488 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:54.588018 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32cac67-61bf-4036-a203-e6cd3e946488 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:54.588504 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32cac67-61bf-4036-a203-e6cd3e946488 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:54.589045 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32cac67-61bf-4036-a203-e6cd3e946488 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:54.651078 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a70113-3e63-4f96-9e0b-f74020de6f18 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:54.664425 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:54.708021 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfccb888-7e12-490e-b4e0-aebc0f09a45f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:54.710443 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfccb888-7e12-490e-b4e0-aebc0f09a45f None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:54.713041 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfccb888-7e12-490e-b4e0-aebc0f09a45f None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:54.713532 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfccb888-7e12-490e-b4e0-aebc0f09a45f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:54.714501 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfccb888-7e12-490e-b4e0-aebc0f09a45f None admin] RBAC: Adding query filter params (role.id=0390f926e7ef4715ae18697033bde90e, scope.project.id=b504a5058bc6474c8cc077dfeec16799, user.id=bcb73dec73464a3abc4b774abb4f416f) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:54.716124 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfccb888-7e12-490e-b4e0-aebc0f09a45f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:54.717835 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfccb888-7e12-490e-b4e0-aebc0f09a45f None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:54.732098 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 344/698] 10.222.0.27 () {56 vars in 1374 bytes} [Sun Sep 11 00:39:54 2022] GET /identity/v3/role_assignments?user.id=bcb73dec73464a3abc4b774abb4f416f&scope.project.id=b504a5058bc6474c8cc077dfeec16799&role.id=0390f926e7ef4715ae18697033bde90e => generated 608 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:54.743155 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c32cac67-61bf-4036-a203-e6cd3e946488 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:54.745440 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c32cac67-61bf-4036-a203-e6cd3e946488 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:54.859648 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 355/699] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:39:54 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 552 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 11 00:39:54.867930 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfccb888-7e12-490e-b4e0-aebc0f09a45f None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:54.983655 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfccb888-7e12-490e-b4e0-aebc0f09a45f None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:55.024264 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc5e8c7-9e8f-4bc7-8ea7-4a3f3ea8fbaa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:55.025295 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc5e8c7-9e8f-4bc7-8ea7-4a3f3ea8fbaa None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:55.025790 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc5e8c7-9e8f-4bc7-8ea7-4a3f3ea8fbaa None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:55.026252 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc5e8c7-9e8f-4bc7-8ea7-4a3f3ea8fbaa None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:55.038826 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dc5e8c7-9e8f-4bc7-8ea7-4a3f3ea8fbaa None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:55.040726 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dc5e8c7-9e8f-4bc7-8ea7-4a3f3ea8fbaa None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:55.056638 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-1dc5e8c7-9e8f-4bc7-8ea7-4a3f3ea8fbaa None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:55.056638 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:55.056638 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:55.056638 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:55.056638 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:55.056638 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:55.056638 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:55.056638 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:55.056638 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:55.056638 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:55.056638 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:55.056638 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:55.056638 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:39:55.056638 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:39:55.056638 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:39:55.056638 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:39:55.057683 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:55.057683 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:55.057683 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:55.057683 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:55.057683 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:55.057683 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:55.057683 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:55.057683 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:55.057683 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:55.057683 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:55.057683 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:55.057683 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:55.057683 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:55.057683 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:55.057683 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:55.057683 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:55.057683 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:39:55.058621 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:39:55.058621 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:55.058621 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:55.060843 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 345/700] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:39:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:55.066432 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c32cac67-61bf-4036-a203-e6cd3e946488 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:55.093655 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:55.183106 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c32cac67-61bf-4036-a203-e6cd3e946488 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:55.219762 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b6e770-1e22-48fd-8987-194c44928777 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:55.220427 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b6e770-1e22-48fd-8987-194c44928777 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:55.220709 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b6e770-1e22-48fd-8987-194c44928777 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:55.220972 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b6e770-1e22-48fd-8987-194c44928777 None admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:55.221620 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2b6e770-1e22-48fd-8987-194c44928777 None admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:55.221922 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2b6e770-1e22-48fd-8987-194c44928777 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:55.223384 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2b6e770-1e22-48fd-8987-194c44928777 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:55.236458 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 356/701] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:39:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:55.242066 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc5e8c7-9e8f-4bc7-8ea7-4a3f3ea8fbaa None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:55.269405 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:55.284525 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbcef7e-10dc-4acf-8ffa-71a191791b67 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:55.284823 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbcef7e-10dc-4acf-8ffa-71a191791b67 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:55.285083 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbcef7e-10dc-4acf-8ffa-71a191791b67 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:55.286684 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 357/702] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:39:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 11 00:39:55.292488 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c3c918a-8ac7-4fbd-9ffb-448e1e0248cf None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:55.293124 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c3c918a-8ac7-4fbd-9ffb-448e1e0248cf None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:55.293587 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c3c918a-8ac7-4fbd-9ffb-448e1e0248cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:55.318919 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-3c3c918a-8ac7-4fbd-9ffb-448e1e0248cf None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:39:55.321136 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-3c3c918a-8ac7-4fbd-9ffb-448e1e0248cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:39:55.368601 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc5e8c7-9e8f-4bc7-8ea7-4a3f3ea8fbaa None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:55.414822 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb044ba6-8f26-4ae0-9048-92662a946d07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:55.415710 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 358/703] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:39:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:39:55.416148 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb044ba6-8f26-4ae0-9048-92662a946d07 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:55.416627 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb044ba6-8f26-4ae0-9048-92662a946d07 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:55.416979 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e015e4-577e-4a68-b771-998839819f50 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:55.417268 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e015e4-577e-4a68-b771-998839819f50 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:55.417541 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e015e4-577e-4a68-b771-998839819f50 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:55.417763 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb044ba6-8f26-4ae0-9048-92662a946d07 None admin] PATH_INFO: `/v3/projects/service` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:55.421341 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 359/704] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:39:55 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:39:55.422471 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb34563-0a8c-4f65-8c8d-02e8c27a04cf None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:55.422756 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb34563-0a8c-4f65-8c8d-02e8c27a04cf None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:55.423074 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb34563-0a8c-4f65-8c8d-02e8c27a04cf None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:55.424304 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 360/705] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:39:55 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:39:55.429805 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22ec73c-531c-4a81-aea6-eb5adddf37dd None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:55.430267 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22ec73c-531c-4a81-aea6-eb5adddf37dd None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:55.430691 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22ec73c-531c-4a81-aea6-eb5adddf37dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:55.437946 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb044ba6-8f26-4ae0-9048-92662a946d07 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:55.440572 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb044ba6-8f26-4ae0-9048-92662a946d07 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:55.450451 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-c22ec73c-531c-4a81-aea6-eb5adddf37dd None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:39:55.451875 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-c22ec73c-531c-4a81-aea6-eb5adddf37dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:39:55.455705 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-bb044ba6-8f26-4ae0-9048-92662a946d07 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:39:55.455705 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:55.455705 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:55.455705 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:55.455705 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:55.455705 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:55.455705 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:55.455705 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:55.455705 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:55.455705 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:55.455705 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:55.455705 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:55.455705 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 11 00:39:55.455705 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 11 00:39:55.455705 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 11 00:39:55.456688 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 11 00:39:55.456688 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:55.456688 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:55.456688 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:55.456688 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:55.456688 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:55.456688 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:55.456688 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:55.456688 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:55.456688 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:55.456688 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:55.456688 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:55.456688 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:55.456688 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:55.456688 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:55.456688 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:55.456688 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:55.457604 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 11 00:39:55.457604 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 11 00:39:55.457604 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 11 00:39:55.457604 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 11 00:39:55.457604 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 11 00:39:55.457604 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 11 00:39:55.457604 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:39:55.457604 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:55.458050 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 346/706] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:39:55 2022] GET /identity/v3/projects/service => generated 88 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:55.459182 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-663f91b9-a0d2-4c82-94e0-72fe5b11e3e7 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:55.459466 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-663f91b9-a0d2-4c82-94e0-72fe5b11e3e7 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:55.459736 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-663f91b9-a0d2-4c82-94e0-72fe5b11e3e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:55.465456 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:55.487434 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-663f91b9-a0d2-4c82-94e0-72fe5b11e3e7 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:39:55.488852 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-663f91b9-a0d2-4c82-94e0-72fe5b11e3e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:39:55.542894 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 361/707] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:39:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 11 00:39:55.544246 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c22ec73c-531c-4a81-aea6-eb5adddf37dd None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:55.591593 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 347/708] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:39:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:39:55.674193 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c22ec73c-531c-4a81-aea6-eb5adddf37dd None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:55.709133 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b0ceaa-1f7e-4683-b428-f0a17bc895f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:55.709817 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b0ceaa-1f7e-4683-b428-f0a17bc895f7 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:55.710169 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b0ceaa-1f7e-4683-b428-f0a17bc895f7 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:55.710601 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b0ceaa-1f7e-4683-b428-f0a17bc895f7 None admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:55.711344 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38b0ceaa-1f7e-4683-b428-f0a17bc895f7 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:55.711649 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38b0ceaa-1f7e-4683-b428-f0a17bc895f7 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:55.713089 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38b0ceaa-1f7e-4683-b428-f0a17bc895f7 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:55.729066 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 362/709] 10.222.0.27 () {56 vars in 1148 bytes} [Sun Sep 11 00:39:55 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:55.732349 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-663f91b9-a0d2-4c82-94e0-72fe5b11e3e7 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:55.734364 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b0ceaa-1f7e-4683-b428-f0a17bc895f7 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:55.758207 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:55.763124 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:55.905650 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b0ceaa-1f7e-4683-b428-f0a17bc895f7 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:55.911679 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-663f91b9-a0d2-4c82-94e0-72fe5b11e3e7 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:55.947369 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16516802-7da8-4f70-afbe-5fc1ec5e32a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:55.947990 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16516802-7da8-4f70-afbe-5fc1ec5e32a7 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:55.948276 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16516802-7da8-4f70-afbe-5fc1ec5e32a7 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:55.948545 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16516802-7da8-4f70-afbe-5fc1ec5e32a7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:55.949274 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16516802-7da8-4f70-afbe-5fc1ec5e32a7 None admin] RBAC: Adding query filter params (role.id=7d9213c1187c4b5182bf8132cf23ea19, scope.project.id=b504a5058bc6474c8cc077dfeec16799, user.id=835def0e6c6d4365a2767051509eb0d6) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:55.949582 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16516802-7da8-4f70-afbe-5fc1ec5e32a7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:55.951156 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16516802-7da8-4f70-afbe-5fc1ec5e32a7 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:55.953713 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f3a8ee-3a70-4746-9368-90005da59986 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:55.954336 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f3a8ee-3a70-4746-9368-90005da59986 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:55.954607 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f3a8ee-3a70-4746-9368-90005da59986 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:55.954896 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f3a8ee-3a70-4746-9368-90005da59986 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:55.957022 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 363/710] 10.222.0.27 () {56 vars in 1374 bytes} [Sun Sep 11 00:39:55 2022] GET /identity/v3/role_assignments?user.id=835def0e6c6d4365a2767051509eb0d6&scope.project.id=b504a5058bc6474c8cc077dfeec16799&role.id=7d9213c1187c4b5182bf8132cf23ea19 => generated 259 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:39:55.958865 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16516802-7da8-4f70-afbe-5fc1ec5e32a7 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:55.967975 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f3a8ee-3a70-4746-9368-90005da59986 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:55.969741 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f3a8ee-3a70-4746-9368-90005da59986 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:55.990901 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-f4f3a8ee-3a70-4746-9368-90005da59986 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:55.990901 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:55.990901 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:55.990901 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:55.990901 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:55.990901 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:55.990901 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:55.990901 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:55.990901 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:55.990901 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:55.990901 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:55.990901 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:55.990901 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:39:55.990901 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:39:55.990901 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:39:55.990901 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:39:55.992715 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:55.992715 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:55.992715 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:55.992715 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:55.992715 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:55.992715 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:55.992715 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:55.992715 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:55.992715 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:55.992715 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:55.992715 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:55.992715 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:55.992715 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:55.992715 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:55.992715 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:55.992715 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:55.992715 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:39:55.993830 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:39:55.993830 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:55.993830 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:55.996472 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 348/711] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:39:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 264 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:55.997542 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f3a8ee-3a70-4746-9368-90005da59986 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:56.046597 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:56.096745 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16516802-7da8-4f70-afbe-5fc1ec5e32a7 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:56.102985 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:56.148979 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-350be729-6cac-4844-9e43-a20cb5e2eba9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:56.150233 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350be729-6cac-4844-9e43-a20cb5e2eba9 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:56.150689 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350be729-6cac-4844-9e43-a20cb5e2eba9 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:56.151125 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350be729-6cac-4844-9e43-a20cb5e2eba9 None admin] PATH_INFO: `/v3/services/network` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:56.152386 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-350be729-6cac-4844-9e43-a20cb5e2eba9 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:56.154323 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-350be729-6cac-4844-9e43-a20cb5e2eba9 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:56.158207 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f3a8ee-3a70-4746-9368-90005da59986 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:56.168779 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-350be729-6cac-4844-9e43-a20cb5e2eba9 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 11 00:39:56.168779 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:56.168779 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 11 00:39:56.168779 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 11 00:39:56.168779 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 11 00:39:56.168779 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 11 00:39:56.168779 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 11 00:39:56.168779 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 11 00:39:56.168779 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 11 00:39:56.168779 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:56.168779 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 11 00:39:56.168779 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:56.168779 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:56.168779 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:56.168779 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:56.168779 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:56.168779 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:56.169812 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:56.169812 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:56.169812 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:56.169812 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:56.169812 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:56.169812 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:56.169812 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 11 00:39:56.169812 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 11 00:39:56.169812 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 11 00:39:56.169812 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 11 00:39:56.169812 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:56.169812 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:56.169812 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:56.169812 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:56.169812 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:56.169812 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:56.170662 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:56.170662 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:56.170662 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:56.170662 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:56.170662 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:56.170662 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:56.170662 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:56.170662 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:56.170662 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:56.170662 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:56.170662 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 11 00:39:56.170662 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 11 00:39:56.170662 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 11 00:39:56.170662 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:56.170662 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 364/712] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:39:55 2022] GET /identity/v3/services/network => generated 88 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:56.172040 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-350be729-6cac-4844-9e43-a20cb5e2eba9 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:56.210731 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb2c4306-33cc-417a-9564-05cd0b1f627f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:56.212282 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb2c4306-33cc-417a-9564-05cd0b1f627f None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:56.213090 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb2c4306-33cc-417a-9564-05cd0b1f627f None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:56.213559 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb2c4306-33cc-417a-9564-05cd0b1f627f None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:56.214337 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb2c4306-33cc-417a-9564-05cd0b1f627f None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:56.217032 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb2c4306-33cc-417a-9564-05cd0b1f627f None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:56.231671 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-bb2c4306-33cc-417a-9564-05cd0b1f627f None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 11 00:39:56.231671 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:56.231671 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 11 00:39:56.231671 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 11 00:39:56.231671 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 11 00:39:56.231671 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 11 00:39:56.231671 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 11 00:39:56.231671 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 11 00:39:56.231671 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 11 00:39:56.231671 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:56.231671 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 11 00:39:56.231671 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:56.231671 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:56.231671 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:56.231671 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:56.231671 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:56.231671 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:56.232693 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:56.232693 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:56.232693 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:56.232693 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:56.232693 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:56.232693 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:56.232693 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 11 00:39:56.232693 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 11 00:39:56.232693 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 11 00:39:56.232693 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 11 00:39:56.232693 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:56.232693 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:56.232693 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:56.232693 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:56.232693 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:56.232693 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:56.233533 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:56.233533 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:56.233533 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:56.233533 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:56.233533 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:56.233533 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:56.233533 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:56.233533 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:56.233533 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:56.233533 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:56.233533 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 11 00:39:56.233533 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 11 00:39:56.233533 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 11 00:39:56.233533 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:56.243169 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 349/713] 10.222.0.27 () {56 vars in 1129 bytes} [Sun Sep 11 00:39:55 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:56.243169 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb2c4306-33cc-417a-9564-05cd0b1f627f None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:56.299578 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-350be729-6cac-4844-9e43-a20cb5e2eba9 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:56.332385 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:56.339920 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d0c673-e23b-47c1-8e5a-b87bb0eceba9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:56.340847 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d0c673-e23b-47c1-8e5a-b87bb0eceba9 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:56.341325 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d0c673-e23b-47c1-8e5a-b87bb0eceba9 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:56.341818 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d0c673-e23b-47c1-8e5a-b87bb0eceba9 None admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:56.342666 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d0c673-e23b-47c1-8e5a-b87bb0eceba9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:56.343492 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d0c673-e23b-47c1-8e5a-b87bb0eceba9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:56.345086 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d0c673-e23b-47c1-8e5a-b87bb0eceba9 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:56.350441 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb2c4306-33cc-417a-9564-05cd0b1f627f None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:56.358718 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 365/714] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:39:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:56.359897 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d0c673-e23b-47c1-8e5a-b87bb0eceba9 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:56.380901 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:56.388942 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82744372-0e1d-4758-a264-f138e1268900 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:56.389552 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82744372-0e1d-4758-a264-f138e1268900 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:56.389824 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82744372-0e1d-4758-a264-f138e1268900 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:56.390093 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82744372-0e1d-4758-a264-f138e1268900 None admin] PATH_INFO: `/v3/services` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:56.390720 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82744372-0e1d-4758-a264-f138e1268900 None admin] RBAC: Adding query filter params (name=network) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:56.391043 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82744372-0e1d-4758-a264-f138e1268900 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:56.392511 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82744372-0e1d-4758-a264-f138e1268900 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:56.402354 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 350/715] 10.222.0.27 () {56 vars in 1112 bytes} [Sun Sep 11 00:39:56 2022] GET /identity/v3/services?name=network => generated 124 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:56.402354 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82744372-0e1d-4758-a264-f138e1268900 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:56.496255 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d0c673-e23b-47c1-8e5a-b87bb0eceba9 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:56.517041 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82744372-0e1d-4758-a264-f138e1268900 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:56.533674 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c946a6-c84d-4503-9d0c-9fd15b04b370 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:56.534287 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c946a6-c84d-4503-9d0c-9fd15b04b370 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:56.534566 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c946a6-c84d-4503-9d0c-9fd15b04b370 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:56.534875 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c946a6-c84d-4503-9d0c-9fd15b04b370 None admin] PATH_INFO: `/v3/services` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:56.535775 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35c946a6-c84d-4503-9d0c-9fd15b04b370 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:56.536080 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35c946a6-c84d-4503-9d0c-9fd15b04b370 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:56.537449 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35c946a6-c84d-4503-9d0c-9fd15b04b370 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:56.547313 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 366/716] 10.222.0.27 () {56 vars in 1124 bytes} [Sun Sep 11 00:39:56 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:56.547313 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c946a6-c84d-4503-9d0c-9fd15b04b370 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:56.572989 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b6b6f18-9155-4d52-ac63-66005168a071 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:56.574062 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6b6f18-9155-4d52-ac63-66005168a071 None admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:56.574497 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6b6f18-9155-4d52-ac63-66005168a071 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:56.575170 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6b6f18-9155-4d52-ac63-66005168a071 None admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:56.576376 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b6b6f18-9155-4d52-ac63-66005168a071 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:56.578475 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b6b6f18-9155-4d52-ac63-66005168a071 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:56.611254 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 351/717] 10.222.0.27 () {60 vars in 1132 bytes} [Sun Sep 11 00:39:56 2022] POST /identity/v3/users => generated 254 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:39:56.611574 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:56.612904 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b6b6f18-9155-4d52-ac63-66005168a071 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:56.716852 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c946a6-c84d-4503-9d0c-9fd15b04b370 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:56.737066 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:56.807220 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eabc7df-4f3b-4feb-adf9-2b7803e9c9e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:56.807962 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eabc7df-4f3b-4feb-adf9-2b7803e9c9e6 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:56.808097 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eabc7df-4f3b-4feb-adf9-2b7803e9c9e6 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:56.808366 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eabc7df-4f3b-4feb-adf9-2b7803e9c9e6 None admin] PATH_INFO: `/v3/services` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:56.809024 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eabc7df-4f3b-4feb-adf9-2b7803e9c9e6 None admin] RBAC: Adding query filter params (type=network) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:56.809298 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eabc7df-4f3b-4feb-adf9-2b7803e9c9e6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:56.810653 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eabc7df-4f3b-4feb-adf9-2b7803e9c9e6 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:56.836135 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 367/718] 10.222.0.27 () {56 vars in 1112 bytes} [Sun Sep 11 00:39:56 2022] GET /identity/v3/services?type=network => generated 124 bytes in 291 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:56.844439 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b6b6f18-9155-4d52-ac63-66005168a071 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:56.913566 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fdb3741-2804-424d-8cc5-7eeb3c09c695 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:56.914601 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fdb3741-2804-424d-8cc5-7eeb3c09c695 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:56.915107 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fdb3741-2804-424d-8cc5-7eeb3c09c695 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:56.915604 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fdb3741-2804-424d-8cc5-7eeb3c09c695 None admin] PATH_INFO: `/v3/services` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:56.916466 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fdb3741-2804-424d-8cc5-7eeb3c09c695 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:56.916980 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fdb3741-2804-424d-8cc5-7eeb3c09c695 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:56.918696 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fdb3741-2804-424d-8cc5-7eeb3c09c695 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:56.929451 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 352/719] 10.222.0.27 () {56 vars in 1124 bytes} [Sun Sep 11 00:39:56 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 317 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:57.417257 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7efc45-0767-47ea-88fc-e9c9aee44fd2 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:57.417607 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7efc45-0767-47ea-88fc-e9c9aee44fd2 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:57.417812 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7efc45-0767-47ea-88fc-e9c9aee44fd2 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:57.419441 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 368/720] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:39:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:39:57.432710 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a41c19-a3f1-40e0-8fd6-7efbf0819094 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:57.433222 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a41c19-a3f1-40e0-8fd6-7efbf0819094 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:57.433222 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a41c19-a3f1-40e0-8fd6-7efbf0819094 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:57.452595 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-79a41c19-a3f1-40e0-8fd6-7efbf0819094 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:39:57.453779 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-79a41c19-a3f1-40e0-8fd6-7efbf0819094 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:39:57.526690 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 353/721] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:39:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:39:57.936420 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7efc45-0767-47ea-88fc-e9c9aee44fd2 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:58.081996 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7efc45-0767-47ea-88fc-e9c9aee44fd2 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:58.112131 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:58.140665 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0dbee5a-4ed4-4943-be67-32cc2d8d40d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:58.141318 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0dbee5a-4ed4-4943-be67-32cc2d8d40d3 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:58.141595 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0dbee5a-4ed4-4943-be67-32cc2d8d40d3 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:58.141864 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0dbee5a-4ed4-4943-be67-32cc2d8d40d3 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:58.163709 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0dbee5a-4ed4-4943-be67-32cc2d8d40d3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:39:58.168639 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-c0dbee5a-4ed4-4943-be67-32cc2d8d40d3 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:39:58.168639 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:58.168639 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:58.168639 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:58.168639 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:58.168639 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:58.168639 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:58.168639 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:58.168639 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:58.168639 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:58.168639 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:58.168639 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:58.168639 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:39:58.168639 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:39:58.168639 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:39:58.168639 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:39:58.169573 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:39:58.169573 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:39:58.169573 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:39:58.169573 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:39:58.169573 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:58.169573 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:58.169573 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:58.169573 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:58.169573 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:58.169573 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:58.169573 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:58.169573 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:58.169573 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:58.169573 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:58.169573 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:58.169573 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:58.170426 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:58.170426 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:58.170426 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:58.170426 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:58.170426 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:39:58.170426 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:39:58.170426 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:39:58.170426 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:39:58.170426 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:39:58.170426 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:39:58.170426 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:39:58.170426 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:58.170426 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 369/722] 10.222.0.27 () {56 vars in 1096 bytes} [Sun Sep 11 00:39:57 2022] GET /identity/v3/roles/admin => generated 83 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:58.177097 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79a41c19-a3f1-40e0-8fd6-7efbf0819094 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:58.235616 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:58.302052 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79a41c19-a3f1-40e0-8fd6-7efbf0819094 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:58.349000 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa77490-9f2e-4b9f-a61c-0b69fe70f086 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:58.349601 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa77490-9f2e-4b9f-a61c-0b69fe70f086 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:58.349881 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa77490-9f2e-4b9f-a61c-0b69fe70f086 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:58.350146 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa77490-9f2e-4b9f-a61c-0b69fe70f086 None admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:58.351059 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fa77490-9f2e-4b9f-a61c-0b69fe70f086 None admin] RBAC: Adding query filter params (name=admin) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:58.351358 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fa77490-9f2e-4b9f-a61c-0b69fe70f086 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:58.352717 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fa77490-9f2e-4b9f-a61c-0b69fe70f086 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:58.361519 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 354/723] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:39:58 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:58.367926 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0dbee5a-4ed4-4943-be67-32cc2d8d40d3 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:58.424859 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-067b8282-9424-4889-b3ed-b78599e8b654 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:58.425213 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-067b8282-9424-4889-b3ed-b78599e8b654 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:58.425404 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-067b8282-9424-4889-b3ed-b78599e8b654 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:58.427001 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 355/724] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:39:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:39:58.433050 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13fab4f6-d904-40da-9f40-095c6c9fce57 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:58.433628 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13fab4f6-d904-40da-9f40-095c6c9fce57 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:58.433628 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13fab4f6-d904-40da-9f40-095c6c9fce57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:58.454514 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-13fab4f6-d904-40da-9f40-095c6c9fce57 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:39:58.455879 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-13fab4f6-d904-40da-9f40-095c6c9fce57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:39:58.546691 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0dbee5a-4ed4-4943-be67-32cc2d8d40d3 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:58.562904 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 356/725] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:39:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 11 00:39:58.598303 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab13fdce-90ac-4d9b-88a7-9ff02c6a946e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:58.598969 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab13fdce-90ac-4d9b-88a7-9ff02c6a946e None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:58.599251 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab13fdce-90ac-4d9b-88a7-9ff02c6a946e None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:58.599523 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab13fdce-90ac-4d9b-88a7-9ff02c6a946e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:58.611083 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab13fdce-90ac-4d9b-88a7-9ff02c6a946e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:58.612548 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab13fdce-90ac-4d9b-88a7-9ff02c6a946e None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:58.633631 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-ab13fdce-90ac-4d9b-88a7-9ff02c6a946e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:58.633631 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:58.633631 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:58.633631 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:58.633631 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:58.633631 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:58.633631 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:58.633631 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:58.633631 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:58.633631 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:58.633631 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:58.633631 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:58.633631 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:39:58.633631 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:39:58.633631 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:39:58.633631 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:39:58.634631 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:58.634631 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:58.634631 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:58.634631 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:58.634631 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:58.634631 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:58.634631 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:58.634631 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:58.634631 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:58.634631 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:58.634631 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:58.634631 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:58.634631 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:58.634631 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:58.634631 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:58.634631 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:58.634631 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:39:58.635585 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:39:58.635585 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:58.635585 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:58.639118 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 370/726] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:39:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 272 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:58.643575 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13fab4f6-d904-40da-9f40-095c6c9fce57 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:58.694591 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:58.779481 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13fab4f6-d904-40da-9f40-095c6c9fce57 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:58.829850 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e7da380-d95f-455b-adbd-a8dec55da61f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:58.830381 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7da380-d95f-455b-adbd-a8dec55da61f None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:58.830659 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7da380-d95f-455b-adbd-a8dec55da61f None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:58.830961 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7da380-d95f-455b-adbd-a8dec55da61f None admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:58.831609 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e7da380-d95f-455b-adbd-a8dec55da61f None admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:58.831910 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e7da380-d95f-455b-adbd-a8dec55da61f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:58.833270 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e7da380-d95f-455b-adbd-a8dec55da61f None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:58.849444 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 357/727] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:39:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:58.855426 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab13fdce-90ac-4d9b-88a7-9ff02c6a946e None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:58.886216 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e7da380-d95f-455b-adbd-a8dec55da61f None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:58.915759 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:59.040517 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab13fdce-90ac-4d9b-88a7-9ff02c6a946e None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:59.062488 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:59.072752 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e7da380-d95f-455b-adbd-a8dec55da61f None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:59.115664 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af64a666-1581-4c06-988c-6eec5438f452 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:59.117091 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af64a666-1581-4c06-988c-6eec5438f452 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:59.117578 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af64a666-1581-4c06-988c-6eec5438f452 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:59.117901 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af64a666-1581-4c06-988c-6eec5438f452 None admin] PATH_INFO: `/v3/users/swift` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:59.133077 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af64a666-1581-4c06-988c-6eec5438f452 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:59.134528 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af64a666-1581-4c06-988c-6eec5438f452 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:59.141493 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09d54804-d457-4654-800f-6df9faab1783 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:59.142127 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d54804-d457-4654-800f-6df9faab1783 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:59.142399 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d54804-d457-4654-800f-6df9faab1783 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:59.142663 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d54804-d457-4654-800f-6df9faab1783 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:59.158101 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-af64a666-1581-4c06-988c-6eec5438f452 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 11 00:39:59.158101 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:59.158101 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:59.158101 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:59.158101 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:59.158101 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:59.158101 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:59.158101 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:59.158101 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:59.158101 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:59.158101 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:59.158101 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:59.158101 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 11 00:39:59.158101 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 11 00:39:59.158101 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 11 00:39:59.158101 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 11 00:39:59.159158 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:59.159158 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:59.159158 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 11 00:39:59.159158 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:39:59.159158 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 11 00:39:59.159158 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:39:59.159158 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:59.159158 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:59.159158 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:59.159158 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:59.159158 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:59.159158 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:59.159158 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:59.159158 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:59.159158 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:59.159158 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:59.159158 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:59.159158 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:59.160878 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:59.160878 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:59.160878 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 11 00:39:59.160878 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 11 00:39:59.160878 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 11 00:39:59.160878 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 11 00:39:59.160878 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 11 00:39:59.160878 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 11 00:39:59.160878 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 11 00:39:59.160878 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:39:59.162168 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 371/728] 10.222.0.27 () {56 vars in 1096 bytes} [Sun Sep 11 00:39:58 2022] GET /identity/v3/users/swift => generated 83 bytes in 306 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:59.165788 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af64a666-1581-4c06-988c-6eec5438f452 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:59.167391 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09d54804-d457-4654-800f-6df9faab1783 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:39:59.173464 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-09d54804-d457-4654-800f-6df9faab1783 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:39:59.173464 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:59.173464 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:59.173464 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:59.173464 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:59.173464 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:59.173464 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:59.173464 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:59.173464 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:59.173464 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:59.173464 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:59.173464 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:59.173464 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:39:59.173464 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:39:59.173464 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:39:59.173464 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:39:59.175679 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:39:59.175679 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:39:59.175679 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:39:59.175679 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:39:59.175679 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:59.175679 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:59.175679 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:59.175679 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:59.175679 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:59.175679 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:59.175679 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:59.175679 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:59.175679 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:59.175679 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:59.175679 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:59.175679 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:59.177775 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:59.177775 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:59.177775 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:59.177775 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:59.177775 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:39:59.177775 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:39:59.177775 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:39:59.177775 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:39:59.177775 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:39:59.177775 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:39:59.177775 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:39:59.177775 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:59.186388 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 358/729] 10.222.0.27 () {56 vars in 1096 bytes} [Sun Sep 11 00:39:58 2022] GET /identity/v3/roles/admin => generated 83 bytes in 291 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:39:59.189245 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09d54804-d457-4654-800f-6df9faab1783 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:59.347160 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:59.360952 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af64a666-1581-4c06-988c-6eec5438f452 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:59.390319 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09d54804-d457-4654-800f-6df9faab1783 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:59.420234 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff4f04cf-a4b6-41fc-83fe-45eabfb48799 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:59.420866 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4f04cf-a4b6-41fc-83fe-45eabfb48799 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:59.421138 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4f04cf-a4b6-41fc-83fe-45eabfb48799 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:59.421406 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4f04cf-a4b6-41fc-83fe-45eabfb48799 None admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:59.422154 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff4f04cf-a4b6-41fc-83fe-45eabfb48799 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:59.422458 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff4f04cf-a4b6-41fc-83fe-45eabfb48799 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:59.431501 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff4f04cf-a4b6-41fc-83fe-45eabfb48799 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:59.465183 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd0ce0bd-b615-4055-af40-a5abe958edf6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:59.465850 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0ce0bd-b615-4055-af40-a5abe958edf6 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:59.466068 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0ce0bd-b615-4055-af40-a5abe958edf6 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:59.468079 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0ce0bd-b615-4055-af40-a5abe958edf6 None admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:59.468079 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd0ce0bd-b615-4055-af40-a5abe958edf6 None admin] RBAC: Adding query filter params (name=admin) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:59.468079 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd0ce0bd-b615-4055-af40-a5abe958edf6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:59.468384 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 372/730] 10.222.0.27 () {56 vars in 1135 bytes} [Sun Sep 11 00:39:59 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 302 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:59.470007 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd0ce0bd-b615-4055-af40-a5abe958edf6 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:59.477646 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff4f04cf-a4b6-41fc-83fe-45eabfb48799 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:59.480343 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 359/731] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:39:59 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 291 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:39:59.491778 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd0ce0bd-b615-4055-af40-a5abe958edf6 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:59.536585 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:59.615971 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff4f04cf-a4b6-41fc-83fe-45eabfb48799 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:59.637387 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd0ce0bd-b615-4055-af40-a5abe958edf6 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:59.656366 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f15aa36-6d68-48db-9333-4471c31bb3e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:59.657312 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f15aa36-6d68-48db-9333-4471c31bb3e1 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:59.657811 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f15aa36-6d68-48db-9333-4471c31bb3e1 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:59.658285 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f15aa36-6d68-48db-9333-4471c31bb3e1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:59.676410 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b6139b6-de00-4df7-a825-abe67376c63e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:59.677041 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6139b6-de00-4df7-a825-abe67376c63e None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:59.677317 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6139b6-de00-4df7-a825-abe67376c63e None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:59.677586 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6139b6-de00-4df7-a825-abe67376c63e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:59.683001 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:59.688774 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b6139b6-de00-4df7-a825-abe67376c63e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:59.690227 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b6139b6-de00-4df7-a825-abe67376c63e None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:59.705207 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-2b6139b6-de00-4df7-a825-abe67376c63e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:59.705207 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:39:59.705207 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:39:59.705207 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:39:59.705207 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:39:59.705207 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:39:59.705207 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:39:59.705207 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:39:59.705207 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:39:59.705207 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:39:59.705207 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:39:59.705207 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:39:59.705207 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:39:59.705207 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:39:59.705207 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:39:59.705207 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:39:59.706384 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:39:59.706384 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:39:59.706384 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:39:59.706384 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:39:59.706384 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:39:59.706384 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:39:59.706384 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:39:59.706384 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:39:59.706384 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:39:59.706384 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:39:59.706384 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:39:59.706384 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:39:59.706384 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:39:59.706384 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:39:59.706384 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:39:59.706384 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:39:59.706384 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:39:59.707348 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:39:59.707348 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:39:59.707348 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:39:59.707348 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 360/732] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:39:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:39:59.707840 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a330c5-2c0d-4fca-b70d-11233d996123 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:59.708036 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a330c5-2c0d-4fca-b70d-11233d996123 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:59.708309 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a330c5-2c0d-4fca-b70d-11233d996123 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:59.709522 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 361/733] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:39:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:39:59.711183 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbcfdf54-7c32-40a4-b425-0b4e80a3bca2 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:59.711661 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbcfdf54-7c32-40a4-b425-0b4e80a3bca2 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:59.711831 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbcfdf54-7c32-40a4-b425-0b4e80a3bca2 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:59.713122 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 362/734] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:39:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:39:59.714299 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323c843f-921b-4233-8a90-014656f4ec62 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:59.714405 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323c843f-921b-4233-8a90-014656f4ec62 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:59.714674 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323c843f-921b-4233-8a90-014656f4ec62 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:59.716131 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 363/735] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:39:59 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:39:59.716790 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323c843f-921b-4233-8a90-014656f4ec62 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:39:59.745459 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f15aa36-6d68-48db-9333-4471c31bb3e1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:59.746957 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f15aa36-6d68-48db-9333-4471c31bb3e1 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:59.787092 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:59.819107 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323c843f-921b-4233-8a90-014656f4ec62 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:39:59.844965 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 373/736] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:39:59 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 368 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 11 00:39:59.846203 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31509cae-2344-40ed-9109-b68746f74f2e None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:59.846487 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31509cae-2344-40ed-9109-b68746f74f2e None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:59.846757 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31509cae-2344-40ed-9109-b68746f74f2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:59.855591 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4f4227f-e192-4caf-af84-d5ed51baac92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:39:59.856221 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4f4227f-e192-4caf-af84-d5ed51baac92 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:59.856498 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4f4227f-e192-4caf-af84-d5ed51baac92 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:59.856766 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4f4227f-e192-4caf-af84-d5ed51baac92 None admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:59.857408 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4f4227f-e192-4caf-af84-d5ed51baac92 None admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:39:59.857706 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4f4227f-e192-4caf-af84-d5ed51baac92 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:39:59.859178 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4f4227f-e192-4caf-af84-d5ed51baac92 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:39:59.863965 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-31509cae-2344-40ed-9109-b68746f74f2e None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:39:59.865146 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-31509cae-2344-40ed-9109-b68746f74f2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:39:59.871706 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 364/737] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:39:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:39:59.872889 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10664dcc-1ab2-4da0-976f-48c32572a23b None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:59.873176 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10664dcc-1ab2-4da0-976f-48c32572a23b None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:59.873444 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10664dcc-1ab2-4da0-976f-48c32572a23b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:59.891361 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-10664dcc-1ab2-4da0-976f-48c32572a23b None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:39:59.892531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-10664dcc-1ab2-4da0-976f-48c32572a23b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:39:59.912309 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:39:59.935693 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 374/738] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:39:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:39:59.937118 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b73da3-cb9d-40c8-b719-fa6bd8623b7b None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:39:59.937405 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b73da3-cb9d-40c8-b719-fa6bd8623b7b None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:39:59.937678 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b73da3-cb9d-40c8-b719-fa6bd8623b7b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:39:59.955120 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-a2b73da3-cb9d-40c8-b719-fa6bd8623b7b None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:39:59.956299 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-a2b73da3-cb9d-40c8-b719-fa6bd8623b7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:39:59.961407 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 365/739] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:39:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:39:59.962079 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10664dcc-1ab2-4da0-976f-48c32572a23b None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:00.036434 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 375/740] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:39:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 11 00:40:00.037382 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b73da3-cb9d-40c8-b719-fa6bd8623b7b None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:00.051960 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:00.103222 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10664dcc-1ab2-4da0-976f-48c32572a23b None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:00.147229 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a37fab5f-f23d-4816-b306-82e570abfe07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:00.148302 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a37fab5f-f23d-4816-b306-82e570abfe07 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:00.148577 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a37fab5f-f23d-4816-b306-82e570abfe07 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:00.148852 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a37fab5f-f23d-4816-b306-82e570abfe07 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:00.171052 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a37fab5f-f23d-4816-b306-82e570abfe07 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:00.172732 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a37fab5f-f23d-4816-b306-82e570abfe07 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:00.194485 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-a37fab5f-f23d-4816-b306-82e570abfe07 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:40:00.194485 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:00.194485 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:00.194485 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:00.194485 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:00.194485 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:00.194485 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:00.194485 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:00.194485 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:00.194485 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:00.194485 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:00.194485 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:00.194485 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:40:00.194485 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:40:00.194485 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:40:00.194485 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:40:00.195700 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:00.195700 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:00.195700 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:00.195700 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:00.195700 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:00.195700 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:00.195700 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:00.195700 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:00.195700 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:00.195700 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:00.195700 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:00.195700 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:00.195700 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:00.195700 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:00.195700 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:00.195700 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:00.195700 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:40:00.196613 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:40:00.196613 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:40:00.196613 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:00.202338 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 366/741] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:39:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:00.205655 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a37fab5f-f23d-4816-b306-82e570abfe07 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:00.258485 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b73da3-cb9d-40c8-b719-fa6bd8623b7b None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:00.269064 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:00.300796 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3411d8df-e40e-4899-8aa5-2c53c83b4069 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:00.301432 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3411d8df-e40e-4899-8aa5-2c53c83b4069 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:00.302128 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3411d8df-e40e-4899-8aa5-2c53c83b4069 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:00.302405 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3411d8df-e40e-4899-8aa5-2c53c83b4069 None admin] PATH_INFO: `/v3/users/nova` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:00.310092 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3411d8df-e40e-4899-8aa5-2c53c83b4069 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:00.311453 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3411d8df-e40e-4899-8aa5-2c53c83b4069 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:00.313884 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a37fab5f-f23d-4816-b306-82e570abfe07 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:00.323817 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-3411d8df-e40e-4899-8aa5-2c53c83b4069 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 11 00:40:00.323817 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:00.323817 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:00.323817 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:00.323817 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:00.323817 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:00.323817 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:00.323817 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:00.323817 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:00.323817 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:00.323817 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:00.323817 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:00.323817 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 11 00:40:00.323817 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 11 00:40:00.323817 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 11 00:40:00.323817 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 11 00:40:00.324768 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:00.324768 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:00.324768 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 11 00:40:00.324768 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:40:00.324768 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 11 00:40:00.324768 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:40:00.324768 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:00.324768 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:00.324768 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:00.324768 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:00.324768 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:00.324768 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:00.324768 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:00.324768 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:00.324768 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:00.324768 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:00.324768 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:00.324768 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:00.325706 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:00.325706 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:00.325706 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 11 00:40:00.325706 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 11 00:40:00.325706 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 11 00:40:00.325706 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 11 00:40:00.325706 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 11 00:40:00.325706 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 11 00:40:00.325706 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 11 00:40:00.325706 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:00.325706 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 376/742] 10.222.0.27 () {56 vars in 1093 bytes} [Sun Sep 11 00:40:00 2022] GET /identity/v3/users/nova => generated 82 bytes in 288 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:00.325706 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3411d8df-e40e-4899-8aa5-2c53c83b4069 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:00.332254 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:00.356285 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4591d4b-9f4b-4001-9f0a-82da0d89a889 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:00.356891 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4591d4b-9f4b-4001-9f0a-82da0d89a889 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:00.357180 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4591d4b-9f4b-4001-9f0a-82da0d89a889 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:00.357451 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4591d4b-9f4b-4001-9f0a-82da0d89a889 None admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:00.358099 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4591d4b-9f4b-4001-9f0a-82da0d89a889 None admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:00.358399 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4591d4b-9f4b-4001-9f0a-82da0d89a889 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:00.359812 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4591d4b-9f4b-4001-9f0a-82da0d89a889 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:00.372564 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 367/743] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:40:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:00.375679 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4591d4b-9f4b-4001-9f0a-82da0d89a889 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:00.427394 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3411d8df-e40e-4899-8aa5-2c53c83b4069 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:00.462586 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb74949-f6d8-43d1-a67d-0387d0a1542c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:00.463631 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb74949-f6d8-43d1-a67d-0387d0a1542c None admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:00.463965 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb74949-f6d8-43d1-a67d-0387d0a1542c None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:00.464363 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb74949-f6d8-43d1-a67d-0387d0a1542c None admin] PATH_INFO: `/v3/services` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:00.465047 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcb74949-f6d8-43d1-a67d-0387d0a1542c None admin] RBAC: Authorizing `identity:create_service()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:00.466501 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcb74949-f6d8-43d1-a67d-0387d0a1542c None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:00.474618 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4591d4b-9f4b-4001-9f0a-82da0d89a889 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:00.485548 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 377/744] 10.222.0.27 () {60 vars in 1141 bytes} [Sun Sep 11 00:40:00 2022] POST /identity/v3/services => generated 251 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:40:00.486342 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb74949-f6d8-43d1-a67d-0387d0a1542c None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:00.518139 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:00.533961 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8ab439-da09-4dbd-8e53-24328f2c6980 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:00.534625 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8ab439-da09-4dbd-8e53-24328f2c6980 None admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:00.534952 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8ab439-da09-4dbd-8e53-24328f2c6980 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:00.535531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8ab439-da09-4dbd-8e53-24328f2c6980 None admin] PATH_INFO: `/v3/services` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:00.536576 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8ab439-da09-4dbd-8e53-24328f2c6980 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:00.538897 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8ab439-da09-4dbd-8e53-24328f2c6980 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:00.557570 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 368/745] 10.222.0.27 () {60 vars in 1141 bytes} [Sun Sep 11 00:40:00 2022] POST /identity/v3/services => generated 240 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:40:00.558488 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8ab439-da09-4dbd-8e53-24328f2c6980 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:00.610672 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:00.629704 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb74949-f6d8-43d1-a67d-0387d0a1542c None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:00.692961 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d4b42c2-2a26-4960-9f78-a735c34647dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:00.693157 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4b42c2-2a26-4960-9f78-a735c34647dc None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:00.693398 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4b42c2-2a26-4960-9f78-a735c34647dc None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:00.693642 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4b42c2-2a26-4960-9f78-a735c34647dc None admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:00.694482 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d4b42c2-2a26-4960-9f78-a735c34647dc None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:00.694815 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d4b42c2-2a26-4960-9f78-a735c34647dc None admin] RBAC: Authorizing `identity:list_users()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:00.697077 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d4b42c2-2a26-4960-9f78-a735c34647dc None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:00.712382 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8ab439-da09-4dbd-8e53-24328f2c6980 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:00.724552 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 378/746] 10.222.0.27 () {56 vars in 1133 bytes} [Sun Sep 11 00:40:00 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:00.725997 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d4b42c2-2a26-4960-9f78-a735c34647dc None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:00.763461 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10b6661f-7244-48e9-866f-9f38f1384a88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:00.764103 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b6661f-7244-48e9-866f-9f38f1384a88 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:00.765103 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b6661f-7244-48e9-866f-9f38f1384a88 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:00.765103 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b6661f-7244-48e9-866f-9f38f1384a88 None admin] PATH_INFO: `/v3/roles/service` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:00.777298 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10b6661f-7244-48e9-866f-9f38f1384a88 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:40:00.782668 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-10b6661f-7244-48e9-866f-9f38f1384a88 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 11 00:40:00.782668 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:00.782668 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:00.782668 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:00.782668 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:00.782668 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:00.782668 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:00.782668 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:00.782668 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:00.782668 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:00.782668 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:00.782668 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:00.782668 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:40:00.782668 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:40:00.782668 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:40:00.782668 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:40:00.783649 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:40:00.783649 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:40:00.783649 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:40:00.783649 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:40:00.783649 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:00.783649 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:00.783649 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:00.783649 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:00.783649 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:00.783649 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:00.783649 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:00.783649 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:00.783649 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:00.783649 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:00.783649 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:00.783649 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:00.784513 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:00.784513 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:00.784513 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:00.784513 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:00.784513 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:40:00.784513 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:40:00.784513 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:40:00.784513 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:40:00.784513 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:40:00.784513 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:40:00.784513 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 11 00:40:00.784513 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:00.785937 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 369/747] 10.222.0.27 () {56 vars in 1102 bytes} [Sun Sep 11 00:40:00 2022] GET /identity/v3/roles/service => generated 85 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:00.790855 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10b6661f-7244-48e9-866f-9f38f1384a88 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:00.837691 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:00.860620 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d4b42c2-2a26-4960-9f78-a735c34647dc None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:00.892038 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:00.905576 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc1da14-7a43-45ab-9090-76b129ec72cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:00.906208 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc1da14-7a43-45ab-9090-76b129ec72cf None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:00.906538 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc1da14-7a43-45ab-9090-76b129ec72cf None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:00.906828 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc1da14-7a43-45ab-9090-76b129ec72cf None admin] PATH_INFO: `/v3/projects/service` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:00.907546 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10b6661f-7244-48e9-866f-9f38f1384a88 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:00.916152 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fc1da14-7a43-45ab-9090-76b129ec72cf None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:00.918152 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fc1da14-7a43-45ab-9090-76b129ec72cf None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:00.932006 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-1fc1da14-7a43-45ab-9090-76b129ec72cf None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:40:00.932006 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:00.932006 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:00.932006 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:00.932006 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:00.932006 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:00.932006 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:00.932006 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:00.932006 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:00.932006 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:00.932006 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:00.932006 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:00.932006 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 11 00:40:00.932006 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 11 00:40:00.932006 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 11 00:40:00.932953 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 11 00:40:00.932953 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:00.932953 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:00.932953 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:00.932953 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:00.932953 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:00.932953 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:00.932953 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:00.932953 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:00.932953 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:00.932953 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:00.932953 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:00.932953 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:00.932953 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:00.932953 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:00.932953 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:00.932953 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:00.933871 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 11 00:40:00.933871 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 11 00:40:00.933871 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 11 00:40:00.933871 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 11 00:40:00.933871 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 11 00:40:00.933871 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 11 00:40:00.933871 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:40:00.933871 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:00.933871 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 379/748] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:40:00 2022] GET /identity/v3/projects/service => generated 88 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:00.934358 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc1da14-7a43-45ab-9090-76b129ec72cf None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:00.954915 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84241813-ed05-4260-aeae-54661fd761a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:00.956151 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84241813-ed05-4260-aeae-54661fd761a4 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:00.957084 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84241813-ed05-4260-aeae-54661fd761a4 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:00.957555 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84241813-ed05-4260-aeae-54661fd761a4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:00.975139 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84241813-ed05-4260-aeae-54661fd761a4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:00.977417 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84241813-ed05-4260-aeae-54661fd761a4 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:00.997219 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-84241813-ed05-4260-aeae-54661fd761a4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:40:00.997219 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:00.997219 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:00.997219 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:00.997219 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:00.997219 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:00.997219 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:00.997219 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:00.997219 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:00.997219 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:00.997219 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:00.997219 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:00.997219 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:40:00.997219 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:40:00.997219 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:40:00.997219 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:40:00.998228 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:00.998228 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:00.998228 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:00.998228 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:00.998228 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:00.998228 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:00.998228 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:00.998228 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:00.998228 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:00.998228 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:00.998228 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:00.998228 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:00.998228 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:00.998228 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:00.998228 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:00.998228 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:00.998228 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:40:00.999186 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:40:00.999186 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:40:00.999186 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:01.000009 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 370/749] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:40:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:01.000788 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84241813-ed05-4260-aeae-54661fd761a4 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:01.059647 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc1da14-7a43-45ab-9090-76b129ec72cf None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:01.096655 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a248d4b-55c4-4b1f-bb12-2b30b86be370 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:01.097344 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a248d4b-55c4-4b1f-bb12-2b30b86be370 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:01.097629 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a248d4b-55c4-4b1f-bb12-2b30b86be370 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:01.097901 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a248d4b-55c4-4b1f-bb12-2b30b86be370 None admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:01.098572 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a248d4b-55c4-4b1f-bb12-2b30b86be370 None admin] RBAC: Adding query filter params (name=service) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:01.098908 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a248d4b-55c4-4b1f-bb12-2b30b86be370 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:01.100878 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a248d4b-55c4-4b1f-bb12-2b30b86be370 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:01.110150 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 380/750] 10.222.0.27 () {56 vars in 1103 bytes} [Sun Sep 11 00:40:00 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:01.111127 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a248d4b-55c4-4b1f-bb12-2b30b86be370 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:01.120114 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:01.141170 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84241813-ed05-4260-aeae-54661fd761a4 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:01.178905 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f42898-fb88-4c36-ab66-d935a1efe750 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:01.179926 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f42898-fb88-4c36-ab66-d935a1efe750 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:01.180504 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f42898-fb88-4c36-ab66-d935a1efe750 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:01.180995 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f42898-fb88-4c36-ab66-d935a1efe750 None admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:01.181924 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7f42898-fb88-4c36-ab66-d935a1efe750 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:01.182421 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7f42898-fb88-4c36-ab66-d935a1efe750 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:01.184286 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7f42898-fb88-4c36-ab66-d935a1efe750 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:01.194466 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:01.199721 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 371/751] 10.222.0.27 () {56 vars in 1148 bytes} [Sun Sep 11 00:40:01 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:01.200910 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f42898-fb88-4c36-ab66-d935a1efe750 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:01.228367 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a248d4b-55c4-4b1f-bb12-2b30b86be370 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:01.266908 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c7a2eb2-0d62-40c6-82e3-ba8e972abac8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:01.267568 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c7a2eb2-0d62-40c6-82e3-ba8e972abac8 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:01.267830 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c7a2eb2-0d62-40c6-82e3-ba8e972abac8 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:01.268039 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c7a2eb2-0d62-40c6-82e3-ba8e972abac8 None admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:01.268677 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c7a2eb2-0d62-40c6-82e3-ba8e972abac8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:01.269265 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c7a2eb2-0d62-40c6-82e3-ba8e972abac8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:01.270630 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c7a2eb2-0d62-40c6-82e3-ba8e972abac8 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:01.287133 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 381/752] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:40:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:01.288712 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c7a2eb2-0d62-40c6-82e3-ba8e972abac8 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:01.325903 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f42898-fb88-4c36-ab66-d935a1efe750 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:01.362463 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:01.365468 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce5c6902-fc84-4416-999b-cb26d9c3e596 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:01.367127 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5c6902-fc84-4416-999b-cb26d9c3e596 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:01.368546 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5c6902-fc84-4416-999b-cb26d9c3e596 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:01.369020 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5c6902-fc84-4416-999b-cb26d9c3e596 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:01.380200 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce5c6902-fc84-4416-999b-cb26d9c3e596 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:01.383101 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce5c6902-fc84-4416-999b-cb26d9c3e596 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:01.399963 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-ce5c6902-fc84-4416-999b-cb26d9c3e596 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:40:01.399963 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:01.399963 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:01.399963 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:01.399963 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:01.399963 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:01.399963 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:01.399963 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:01.399963 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:01.399963 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:01.399963 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:01.399963 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:01.399963 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:40:01.399963 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:40:01.399963 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:40:01.399963 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:40:01.400969 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:01.400969 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:01.400969 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:01.400969 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:01.400969 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:01.400969 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:01.400969 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:01.400969 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:01.400969 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:01.400969 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:01.400969 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:01.400969 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:01.400969 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:01.400969 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:01.400969 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:01.400969 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:01.400969 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:40:01.401955 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:40:01.401955 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:40:01.401955 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:01.404244 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 372/753] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:40:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:01.405907 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce5c6902-fc84-4416-999b-cb26d9c3e596 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:01.409019 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c7a2eb2-0d62-40c6-82e3-ba8e972abac8 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:01.448158 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3c6fc18-81dc-4c6b-80ba-9b5794482ae7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:01.448764 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c6fc18-81dc-4c6b-80ba-9b5794482ae7 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:01.449037 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c6fc18-81dc-4c6b-80ba-9b5794482ae7 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:01.449303 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c6fc18-81dc-4c6b-80ba-9b5794482ae7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:01.450089 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3c6fc18-81dc-4c6b-80ba-9b5794482ae7 None admin] RBAC: Adding query filter params (role.id=7d9213c1187c4b5182bf8132cf23ea19, scope.project.id=b504a5058bc6474c8cc077dfeec16799, user.id=bcb73dec73464a3abc4b774abb4f416f) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:01.450396 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3c6fc18-81dc-4c6b-80ba-9b5794482ae7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:01.452025 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3c6fc18-81dc-4c6b-80ba-9b5794482ae7 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:01.458622 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 382/754] 10.222.0.27 () {56 vars in 1374 bytes} [Sun Sep 11 00:40:01 2022] GET /identity/v3/role_assignments?user.id=bcb73dec73464a3abc4b774abb4f416f&scope.project.id=b504a5058bc6474c8cc077dfeec16799&role.id=7d9213c1187c4b5182bf8132cf23ea19 => generated 259 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:01.460891 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3c6fc18-81dc-4c6b-80ba-9b5794482ae7 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:01.489321 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:01.602068 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce5c6902-fc84-4416-999b-cb26d9c3e596 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:01.620943 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3c6fc18-81dc-4c6b-80ba-9b5794482ae7 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:01.653473 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:01.665953 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2190b2ff-5703-4818-990e-5750a32ce2d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:01.666769 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2190b2ff-5703-4818-990e-5750a32ce2d4 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:01.667246 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2190b2ff-5703-4818-990e-5750a32ce2d4 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:01.667662 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2190b2ff-5703-4818-990e-5750a32ce2d4 None admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:01.667778 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919ee26e-3dd2-4c6e-a7ce-931190b9b291 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:01.668387 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919ee26e-3dd2-4c6e-a7ce-931190b9b291 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:01.668667 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919ee26e-3dd2-4c6e-a7ce-931190b9b291 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:01.668864 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2190b2ff-5703-4818-990e-5750a32ce2d4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:01.668970 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919ee26e-3dd2-4c6e-a7ce-931190b9b291 None admin] PATH_INFO: `/v3/projects/service` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:01.669434 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2190b2ff-5703-4818-990e-5750a32ce2d4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:01.670935 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2190b2ff-5703-4818-990e-5750a32ce2d4 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:01.676132 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-919ee26e-3dd2-4c6e-a7ce-931190b9b291 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:01.677866 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-919ee26e-3dd2-4c6e-a7ce-931190b9b291 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:01.684569 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 383/755] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:40:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:01.689866 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2190b2ff-5703-4818-990e-5750a32ce2d4 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:01.695158 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-919ee26e-3dd2-4c6e-a7ce-931190b9b291 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:40:01.695158 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:01.695158 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:01.695158 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:01.695158 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:01.695158 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:01.695158 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:01.695158 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:01.695158 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:01.695158 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:01.695158 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:01.695158 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:01.695158 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 11 00:40:01.695158 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 11 00:40:01.695158 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 11 00:40:01.696089 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 11 00:40:01.696089 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:01.696089 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:01.696089 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:01.696089 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:01.696089 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:01.696089 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:01.696089 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:01.696089 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:01.696089 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:01.696089 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:01.696089 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:01.696089 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:01.696089 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:01.696089 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:01.696089 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:01.696089 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:01.697061 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 11 00:40:01.697061 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 11 00:40:01.697061 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 11 00:40:01.697061 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 11 00:40:01.697061 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 11 00:40:01.697061 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 11 00:40:01.697061 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:40:01.697061 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:01.699429 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 373/756] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:40:01 2022] GET /identity/v3/projects/service => generated 88 bytes in 294 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:01.707775 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919ee26e-3dd2-4c6e-a7ce-931190b9b291 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:01.799147 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2190b2ff-5703-4818-990e-5750a32ce2d4 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:01.841811 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd24832f-713e-4857-92db-8c5f3865a484 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:01.842453 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd24832f-713e-4857-92db-8c5f3865a484 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:01.842730 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd24832f-713e-4857-92db-8c5f3865a484 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:01.843030 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd24832f-713e-4857-92db-8c5f3865a484 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:01.863470 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd24832f-713e-4857-92db-8c5f3865a484 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:01.864931 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd24832f-713e-4857-92db-8c5f3865a484 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:01.888359 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-fd24832f-713e-4857-92db-8c5f3865a484 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 11 00:40:01.888359 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:01.888359 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:01.888359 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:01.888359 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:01.888359 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:01.888359 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:01.888359 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:01.888359 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:01.888359 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:01.888359 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:01.888359 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:01.888359 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 11 00:40:01.888359 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 11 00:40:01.888359 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 11 00:40:01.888359 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 11 00:40:01.889406 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:01.889406 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:01.889406 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 11 00:40:01.889406 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:40:01.889406 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 11 00:40:01.889406 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:40:01.889406 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:01.889406 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:01.889406 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:01.889406 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:01.889406 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:01.889406 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:01.889406 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:01.889406 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:01.889406 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:01.889406 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:01.889406 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:01.889406 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:01.890372 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:01.890372 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:01.890372 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 11 00:40:01.890372 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 11 00:40:01.890372 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 11 00:40:01.890372 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 11 00:40:01.890372 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 11 00:40:01.890372 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 11 00:40:01.890372 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 11 00:40:01.890372 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:01.894098 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 384/757] 10.222.0.27 () {56 vars in 1117 bytes} [Sun Sep 11 00:40:01 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:01.896654 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919ee26e-3dd2-4c6e-a7ce-931190b9b291 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:01.899756 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd24832f-713e-4857-92db-8c5f3865a484 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:01.912027 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:01.937514 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fb7a68a-f1d1-4bfd-a08f-b77f0bb0b105 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:01.938116 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb7a68a-f1d1-4bfd-a08f-b77f0bb0b105 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:01.938394 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb7a68a-f1d1-4bfd-a08f-b77f0bb0b105 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:01.938666 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb7a68a-f1d1-4bfd-a08f-b77f0bb0b105 None admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:01.939390 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fb7a68a-f1d1-4bfd-a08f-b77f0bb0b105 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:01.940203 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fb7a68a-f1d1-4bfd-a08f-b77f0bb0b105 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:01.941808 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fb7a68a-f1d1-4bfd-a08f-b77f0bb0b105 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:01.957317 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 374/758] 10.222.0.27 () {56 vars in 1148 bytes} [Sun Sep 11 00:40:01 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:01.965407 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fb7a68a-f1d1-4bfd-a08f-b77f0bb0b105 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:02.007816 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:02.086545 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd24832f-713e-4857-92db-8c5f3865a484 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:02.153053 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fb7a68a-f1d1-4bfd-a08f-b77f0bb0b105 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:02.164186 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2480ff21-832c-4804-baba-01b636324f5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:02.164800 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2480ff21-832c-4804-baba-01b636324f5f None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:02.165084 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2480ff21-832c-4804-baba-01b636324f5f None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:02.165357 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2480ff21-832c-4804-baba-01b636324f5f None admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:02.166093 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2480ff21-832c-4804-baba-01b636324f5f None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:02.166399 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2480ff21-832c-4804-baba-01b636324f5f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:02.167956 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2480ff21-832c-4804-baba-01b636324f5f None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:02.198730 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 385/759] 10.222.0.27 () {56 vars in 1149 bytes} [Sun Sep 11 00:40:01 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 299 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:02.204032 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2480ff21-832c-4804-baba-01b636324f5f None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:02.207655 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fcc6400-b06e-45bc-9408-fac2675f0490 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:02.208323 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fcc6400-b06e-45bc-9408-fac2675f0490 None admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:02.208599 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fcc6400-b06e-45bc-9408-fac2675f0490 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:02.209031 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fcc6400-b06e-45bc-9408-fac2675f0490 None admin] PATH_INFO: `/v3/projects/b504a5058bc6474c8cc077dfeec16799/users/835def0e6c6d4365a2767051509eb0d6/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:02.211865 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fcc6400-b06e-45bc-9408-fac2675f0490 None admin] RBAC: Authorizing `identity:create_grant(project_id=b504a5058bc6474c8cc077dfeec16799, user_id=835def0e6c6d4365a2767051509eb0d6, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:02.213337 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fcc6400-b06e-45bc-9408-fac2675f0490 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:02.243537 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 375/760] 10.222.0.27 () {58 vars in 1439 bytes} [Sun Sep 11 00:40:01 2022] PUT /identity/v3/projects/b504a5058bc6474c8cc077dfeec16799/users/835def0e6c6d4365a2767051509eb0d6/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 278 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:40:02.338026 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2480ff21-832c-4804-baba-01b636324f5f None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:02.347244 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:02.384700 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc7ae4c-c4b9-4d2b-be63-3215b6882441 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:02.385322 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc7ae4c-c4b9-4d2b-be63-3215b6882441 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:02.385598 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc7ae4c-c4b9-4d2b-be63-3215b6882441 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:02.385866 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc7ae4c-c4b9-4d2b-be63-3215b6882441 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:02.478828 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc7ae4c-c4b9-4d2b-be63-3215b6882441 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:02.480303 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc7ae4c-c4b9-4d2b-be63-3215b6882441 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:02.592662 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 386/761] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:40:02 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 389 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 11 00:40:02.598055 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fcc6400-b06e-45bc-9408-fac2675f0490 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:02.640801 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:02.726975 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fcc6400-b06e-45bc-9408-fac2675f0490 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:02.765187 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767d45f1-0139-494b-bdf3-2fdfdf21bc43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:02.765810 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767d45f1-0139-494b-bdf3-2fdfdf21bc43 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:02.766081 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767d45f1-0139-494b-bdf3-2fdfdf21bc43 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:02.766347 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767d45f1-0139-494b-bdf3-2fdfdf21bc43 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:02.776846 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-767d45f1-0139-494b-bdf3-2fdfdf21bc43 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:02.778286 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-767d45f1-0139-494b-bdf3-2fdfdf21bc43 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:02.793968 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-767d45f1-0139-494b-bdf3-2fdfdf21bc43 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:40:02.793968 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:02.793968 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:02.793968 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:02.793968 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:02.793968 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:02.793968 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:02.793968 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:02.793968 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:02.793968 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:02.793968 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:02.793968 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:02.793968 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:40:02.793968 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:40:02.793968 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:40:02.793968 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:40:02.794982 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:02.794982 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:02.794982 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:02.794982 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:02.794982 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:02.794982 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:02.794982 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:02.794982 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:02.794982 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:02.794982 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:02.794982 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:02.794982 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:02.794982 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:02.794982 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:02.794982 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:02.794982 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:02.794982 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:40:02.795895 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:40:02.795895 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:40:02.795895 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:02.795895 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 376/762] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:40:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:02.804816 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc7ae4c-c4b9-4d2b-be63-3215b6882441 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:02.813224 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:02.917514 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc7ae4c-c4b9-4d2b-be63-3215b6882441 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:02.954940 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b6e51f0-5ff2-490c-b65a-15fadeb4d9fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:02.955601 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6e51f0-5ff2-490c-b65a-15fadeb4d9fc None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:02.955880 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6e51f0-5ff2-490c-b65a-15fadeb4d9fc None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:02.956212 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6e51f0-5ff2-490c-b65a-15fadeb4d9fc None admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:02.956858 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b6e51f0-5ff2-490c-b65a-15fadeb4d9fc None admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:02.957154 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b6e51f0-5ff2-490c-b65a-15fadeb4d9fc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:02.958514 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b6e51f0-5ff2-490c-b65a-15fadeb4d9fc None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:02.971016 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 387/763] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:40:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:02.979909 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767d45f1-0139-494b-bdf3-2fdfdf21bc43 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:03.111182 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:03.134335 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767d45f1-0139-494b-bdf3-2fdfdf21bc43 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:03.151126 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c764a3-31cd-4c69-850f-8500705e48e4 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:03.151459 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c764a3-31cd-4c69-850f-8500705e48e4 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:03.151663 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c764a3-31cd-4c69-850f-8500705e48e4 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:03.153326 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 388/764] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:40:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 11 00:40:03.159350 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1955f0-bbc7-4101-9e37-b9f9a0dd6187 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:03.159618 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1955f0-bbc7-4101-9e37-b9f9a0dd6187 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:03.159882 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1955f0-bbc7-4101-9e37-b9f9a0dd6187 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:03.182235 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-4b1955f0-bbc7-4101-9e37-b9f9a0dd6187 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:40:03.183453 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-4b1955f0-bbc7-4101-9e37-b9f9a0dd6187 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:40:03.185341 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00444e91-21d6-42a5-9342-df7326c2ac16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:03.185980 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00444e91-21d6-42a5-9342-df7326c2ac16 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:03.186255 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00444e91-21d6-42a5-9342-df7326c2ac16 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:03.186521 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00444e91-21d6-42a5-9342-df7326c2ac16 None admin] PATH_INFO: `/v3/projects/service` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:03.200052 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00444e91-21d6-42a5-9342-df7326c2ac16 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:03.201529 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00444e91-21d6-42a5-9342-df7326c2ac16 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:03.227821 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-00444e91-21d6-42a5-9342-df7326c2ac16 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:40:03.227821 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:03.227821 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:03.227821 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:03.227821 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:03.227821 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:03.227821 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:03.227821 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:03.227821 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:03.227821 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:03.227821 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:03.227821 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:03.227821 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 11 00:40:03.227821 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 11 00:40:03.227821 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 11 00:40:03.229941 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 11 00:40:03.229941 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:03.229941 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:03.229941 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:03.229941 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:03.229941 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:03.229941 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:03.229941 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:03.229941 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:03.229941 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:03.229941 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:03.229941 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:03.229941 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:03.229941 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:03.229941 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:03.229941 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:03.229941 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:03.232164 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 11 00:40:03.232164 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 11 00:40:03.232164 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 11 00:40:03.232164 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 11 00:40:03.232164 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 11 00:40:03.232164 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 11 00:40:03.232164 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:40:03.232164 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:03.232164 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 377/765] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:40:02 2022] GET /identity/v3/projects/service => generated 88 bytes in 250 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:03.234588 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00444e91-21d6-42a5-9342-df7326c2ac16 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:03.272014 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:03.286300 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 389/766] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:40:03 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:40:03.321071 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83713ca0-0393-4ef4-b03c-071e0b300bbe None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:03.321381 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83713ca0-0393-4ef4-b03c-071e0b300bbe None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:03.321662 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83713ca0-0393-4ef4-b03c-071e0b300bbe None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:03.323431 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 390/767] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:40:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:40:03.328542 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5769689-67f7-4b2a-8ec7-a3ce2ea2707d None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:03.328822 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5769689-67f7-4b2a-8ec7-a3ce2ea2707d None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:03.329092 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5769689-67f7-4b2a-8ec7-a3ce2ea2707d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:03.348327 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-e5769689-67f7-4b2a-8ec7-a3ce2ea2707d None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:40:03.349520 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-e5769689-67f7-4b2a-8ec7-a3ce2ea2707d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:40:03.380915 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00444e91-21d6-42a5-9342-df7326c2ac16 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:03.425882 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 391/768] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:40:03 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 11 00:40:03.434475 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6b37f11-c08a-4c51-8f6c-c94e2d0209ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:03.435423 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b37f11-c08a-4c51-8f6c-c94e2d0209ca None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:03.435911 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b37f11-c08a-4c51-8f6c-c94e2d0209ca None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:03.436388 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b37f11-c08a-4c51-8f6c-c94e2d0209ca None admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:03.437297 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6b37f11-c08a-4c51-8f6c-c94e2d0209ca None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:03.440527 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6b37f11-c08a-4c51-8f6c-c94e2d0209ca None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:03.443236 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6b37f11-c08a-4c51-8f6c-c94e2d0209ca None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:03.466734 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 378/769] 10.222.0.27 () {56 vars in 1148 bytes} [Sun Sep 11 00:40:03 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:40:03.472525 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5769689-67f7-4b2a-8ec7-a3ce2ea2707d None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:03.617464 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6b37f11-c08a-4c51-8f6c-c94e2d0209ca None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:03.618016 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5769689-67f7-4b2a-8ec7-a3ce2ea2707d None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:03.623798 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:03.655757 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:03.672948 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a440be8-805b-4443-90f6-9f421f54bd64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:03.673569 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a440be8-805b-4443-90f6-9f421f54bd64 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:03.673850 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a440be8-805b-4443-90f6-9f421f54bd64 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:03.674124 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a440be8-805b-4443-90f6-9f421f54bd64 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:03.675080 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a440be8-805b-4443-90f6-9f421f54bd64 None admin] RBAC: Adding query filter params (role.id=0390f926e7ef4715ae18697033bde90e, scope.project.id=b504a5058bc6474c8cc077dfeec16799, user.id=9fc4274836844ae4ad5d16d74c36f671) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:03.675285 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a440be8-805b-4443-90f6-9f421f54bd64 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:03.676909 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a440be8-805b-4443-90f6-9f421f54bd64 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:03.691269 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 392/770] 10.222.0.27 () {56 vars in 1374 bytes} [Sun Sep 11 00:40:03 2022] GET /identity/v3/role_assignments?user.id=9fc4274836844ae4ad5d16d74c36f671&scope.project.id=b504a5058bc6474c8cc077dfeec16799&role.id=0390f926e7ef4715ae18697033bde90e => generated 259 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:03.742827 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6b37f11-c08a-4c51-8f6c-c94e2d0209ca None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:03.781152 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66fd7666-e50c-48aa-9f6e-42e4eabb7df8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:03.782148 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66fd7666-e50c-48aa-9f6e-42e4eabb7df8 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:03.782622 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66fd7666-e50c-48aa-9f6e-42e4eabb7df8 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:03.783134 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66fd7666-e50c-48aa-9f6e-42e4eabb7df8 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:03.783921 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66fd7666-e50c-48aa-9f6e-42e4eabb7df8 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:03.785758 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66fd7666-e50c-48aa-9f6e-42e4eabb7df8 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:03.795938 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-66fd7666-e50c-48aa-9f6e-42e4eabb7df8 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 11 00:40:03.795938 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:03.795938 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 11 00:40:03.795938 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 11 00:40:03.795938 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 11 00:40:03.795938 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 11 00:40:03.795938 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 11 00:40:03.795938 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 11 00:40:03.795938 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 11 00:40:03.795938 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:03.795938 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 11 00:40:03.795938 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:03.795938 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:03.795938 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:03.795938 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:03.795938 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:03.795938 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:03.797011 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:03.797011 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:03.797011 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:03.797011 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:03.797011 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:03.797011 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:03.797011 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 11 00:40:03.797011 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 11 00:40:03.797011 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 11 00:40:03.797011 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 11 00:40:03.797011 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:03.797011 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:03.797011 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:03.797011 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:03.797011 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:03.797011 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:03.797878 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:03.797878 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:03.797878 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:03.797878 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:03.797878 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:03.797878 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:03.797878 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:03.797878 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:03.797878 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:03.797878 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:03.797878 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 11 00:40:03.797878 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 11 00:40:03.797878 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 11 00:40:03.797878 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:03.800834 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 379/771] 10.222.0.27 () {56 vars in 1129 bytes} [Sun Sep 11 00:40:03 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:03.807610 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a440be8-805b-4443-90f6-9f421f54bd64 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:03.823496 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66fd7666-e50c-48aa-9f6e-42e4eabb7df8 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:03.977596 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:04.026491 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66fd7666-e50c-48aa-9f6e-42e4eabb7df8 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:04.084968 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e9ec25-846d-4ed1-ab04-d6947981ce13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:04.088693 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e9ec25-846d-4ed1-ab04-d6947981ce13 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:04.088693 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e9ec25-846d-4ed1-ab04-d6947981ce13 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:04.088693 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e9ec25-846d-4ed1-ab04-d6947981ce13 None admin] PATH_INFO: `/v3/services/network` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:04.088693 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0e9ec25-846d-4ed1-ab04-d6947981ce13 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:04.088693 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0e9ec25-846d-4ed1-ab04-d6947981ce13 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:04.101720 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-e0e9ec25-846d-4ed1-ab04-d6947981ce13 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 11 00:40:04.101720 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:04.101720 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 11 00:40:04.101720 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 11 00:40:04.101720 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 11 00:40:04.101720 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 11 00:40:04.101720 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 11 00:40:04.101720 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 11 00:40:04.101720 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 11 00:40:04.101720 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:04.101720 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 11 00:40:04.101720 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:04.101720 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:04.101720 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:04.101720 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:04.101720 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:04.101720 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:04.102759 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:04.102759 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:04.102759 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:04.102759 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:04.102759 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:04.102759 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:04.102759 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 11 00:40:04.102759 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 11 00:40:04.102759 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 11 00:40:04.102759 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 11 00:40:04.102759 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:04.102759 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:04.102759 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:04.102759 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:04.102759 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:04.102759 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:04.103642 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:04.103642 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:04.103642 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:04.103642 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:04.103642 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:04.103642 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:04.103642 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:04.103642 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:04.103642 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:04.103642 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:04.103642 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 11 00:40:04.103642 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 11 00:40:04.103642 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 11 00:40:04.103642 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:04.103642 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 380/772] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:40:03 2022] GET /identity/v3/services/network => generated 88 bytes in 278 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:40:04.104458 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8789eb58-4a14-4dd6-b3a4-c6a58a1188fa None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:04.104458 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8789eb58-4a14-4dd6-b3a4-c6a58a1188fa None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:04.104458 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8789eb58-4a14-4dd6-b3a4-c6a58a1188fa None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:04.104458 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 381/773] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:40:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:40:04.106309 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a440be8-805b-4443-90f6-9f421f54bd64 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:04.111998 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8789eb58-4a14-4dd6-b3a4-c6a58a1188fa None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:04.148345 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4062171-c81b-467d-8d7b-87c1697d8e99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:04.149064 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4062171-c81b-467d-8d7b-87c1697d8e99 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:04.149347 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4062171-c81b-467d-8d7b-87c1697d8e99 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:04.149620 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4062171-c81b-467d-8d7b-87c1697d8e99 None admin] PATH_INFO: `/v3/services` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:04.150361 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4062171-c81b-467d-8d7b-87c1697d8e99 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:04.150665 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4062171-c81b-467d-8d7b-87c1697d8e99 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:04.152162 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4062171-c81b-467d-8d7b-87c1697d8e99 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:04.158346 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 393/774] 10.222.0.27 () {56 vars in 1124 bytes} [Sun Sep 11 00:40:03 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 351 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:04.160079 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c6775d-73de-4609-bb2d-bf028e3a3a45 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:04.160628 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c6775d-73de-4609-bb2d-bf028e3a3a45 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:04.160901 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c6775d-73de-4609-bb2d-bf028e3a3a45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:04.182508 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-39c6775d-73de-4609-bb2d-bf028e3a3a45 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:40:04.183752 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-39c6775d-73de-4609-bb2d-bf028e3a3a45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:40:04.226337 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8789eb58-4a14-4dd6-b3a4-c6a58a1188fa None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:04.242607 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:04.259264 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:04.270810 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 394/775] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:40:04 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:40:04.270810 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c6775d-73de-4609-bb2d-bf028e3a3a45 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:04.274020 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d3408b1-714d-44ee-a397-5de54359bf40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:04.274925 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3408b1-714d-44ee-a397-5de54359bf40 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:04.275421 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3408b1-714d-44ee-a397-5de54359bf40 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:04.275887 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3408b1-714d-44ee-a397-5de54359bf40 None admin] PATH_INFO: `/v3/services` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:04.276713 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d3408b1-714d-44ee-a397-5de54359bf40 None admin] RBAC: Adding query filter params (name=network) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:04.277256 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d3408b1-714d-44ee-a397-5de54359bf40 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:04.278911 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d3408b1-714d-44ee-a397-5de54359bf40 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:04.292592 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 382/776] 10.222.0.27 () {56 vars in 1112 bytes} [Sun Sep 11 00:40:04 2022] GET /identity/v3/services?name=network => generated 124 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:04.298250 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d3408b1-714d-44ee-a397-5de54359bf40 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:04.377127 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c6775d-73de-4609-bb2d-bf028e3a3a45 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:04.416475 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9532dc6-7a23-4c69-9e1e-0b1e7ca1fdec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:04.417257 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9532dc6-7a23-4c69-9e1e-0b1e7ca1fdec None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:04.417540 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9532dc6-7a23-4c69-9e1e-0b1e7ca1fdec None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:04.417880 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9532dc6-7a23-4c69-9e1e-0b1e7ca1fdec None admin] PATH_INFO: `/v3/services` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:04.418579 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9532dc6-7a23-4c69-9e1e-0b1e7ca1fdec None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:04.418911 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9532dc6-7a23-4c69-9e1e-0b1e7ca1fdec None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:04.420653 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9532dc6-7a23-4c69-9e1e-0b1e7ca1fdec None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:04.426340 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 395/777] 10.222.0.27 () {56 vars in 1124 bytes} [Sun Sep 11 00:40:04 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:04.434066 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9532dc6-7a23-4c69-9e1e-0b1e7ca1fdec None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:04.498136 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d3408b1-714d-44ee-a397-5de54359bf40 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:04.527132 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:04.559271 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc38ef3-7984-41f1-ac9d-0567cd713bcf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:04.560353 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc38ef3-7984-41f1-ac9d-0567cd713bcf None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:04.560837 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc38ef3-7984-41f1-ac9d-0567cd713bcf None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:04.561310 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc38ef3-7984-41f1-ac9d-0567cd713bcf None admin] PATH_INFO: `/v3/services` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:04.562174 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc38ef3-7984-41f1-ac9d-0567cd713bcf None admin] RBAC: Adding query filter params (type=network) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:04.562690 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc38ef3-7984-41f1-ac9d-0567cd713bcf None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:04.564290 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc38ef3-7984-41f1-ac9d-0567cd713bcf None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:04.570286 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 383/778] 10.222.0.27 () {56 vars in 1112 bytes} [Sun Sep 11 00:40:04 2022] GET /identity/v3/services?type=network => generated 350 bytes in 272 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:40:04.582150 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc38ef3-7984-41f1-ac9d-0567cd713bcf None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:04.605238 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9532dc6-7a23-4c69-9e1e-0b1e7ca1fdec None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:04.640509 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa62957-a535-4689-9ea0-0317896ddca1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:04.641282 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa62957-a535-4689-9ea0-0317896ddca1 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:04.641649 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa62957-a535-4689-9ea0-0317896ddca1 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:04.642010 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa62957-a535-4689-9ea0-0317896ddca1 None admin] PATH_INFO: `/v3/endpoints` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:04.642814 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aa62957-a535-4689-9ea0-0317896ddca1 None admin] RBAC: Adding query filter params (interface=public, service_id=19986872fa9a4a65873e71446b4f5b81, region_id=RegionOne) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:04.643299 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aa62957-a535-4689-9ea0-0317896ddca1 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:04.644797 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aa62957-a535-4689-9ea0-0317896ddca1 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:04.653106 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 396/779] 10.222.0.27 () {56 vars in 1251 bytes} [Sun Sep 11 00:40:04 2022] GET /identity/v3/endpoints?service_id=19986872fa9a4a65873e71446b4f5b81&interface=public®ion_id=RegionOne => generated 194 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:04.661519 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa62957-a535-4689-9ea0-0317896ddca1 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:04.680276 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:04.735678 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc38ef3-7984-41f1-ac9d-0567cd713bcf None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:04.774839 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef83ef5-e996-4fd9-926c-db9883723270 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:04.775841 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef83ef5-e996-4fd9-926c-db9883723270 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:04.776331 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef83ef5-e996-4fd9-926c-db9883723270 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:04.776812 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef83ef5-e996-4fd9-926c-db9883723270 None admin] PATH_INFO: `/v3/endpoints` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:04.777773 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ef83ef5-e996-4fd9-926c-db9883723270 None admin] RBAC: Adding query filter params (interface=public, service_id=be35df37167d46dda07f61aced2f83c8, region_id=RegionOne) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:04.778347 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ef83ef5-e996-4fd9-926c-db9883723270 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:04.779969 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ef83ef5-e996-4fd9-926c-db9883723270 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:04.789801 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 384/780] 10.222.0.27 () {56 vars in 1251 bytes} [Sun Sep 11 00:40:04 2022] GET /identity/v3/endpoints?service_id=be35df37167d46dda07f61aced2f83c8&interface=public®ion_id=RegionOne => generated 194 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:40:04.791005 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef83ef5-e996-4fd9-926c-db9883723270 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:04.815592 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa62957-a535-4689-9ea0-0317896ddca1 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:04.830447 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:04.853567 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d34cb38f-397e-4205-b477-92c1cece53bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:04.854180 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d34cb38f-397e-4205-b477-92c1cece53bb None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:04.854453 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d34cb38f-397e-4205-b477-92c1cece53bb None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:04.854718 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d34cb38f-397e-4205-b477-92c1cece53bb None admin] PATH_INFO: `/v3/services` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:04.855568 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d34cb38f-397e-4205-b477-92c1cece53bb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:04.856910 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d34cb38f-397e-4205-b477-92c1cece53bb None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:04.862638 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 397/781] 10.222.0.27 () {56 vars in 1087 bytes} [Sun Sep 11 00:40:04 2022] GET /identity/v3/services => generated 772 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:04.863483 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d34cb38f-397e-4205-b477-92c1cece53bb None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:04.905917 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef83ef5-e996-4fd9-926c-db9883723270 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:04.933258 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:04.946240 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-778e0f02-d60a-4f42-a7a7-e2566daacb46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:04.946863 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-778e0f02-d60a-4f42-a7a7-e2566daacb46 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:04.947245 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-778e0f02-d60a-4f42-a7a7-e2566daacb46 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:04.947626 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-778e0f02-d60a-4f42-a7a7-e2566daacb46 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:04.960353 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-778e0f02-d60a-4f42-a7a7-e2566daacb46 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:40:04.965878 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-778e0f02-d60a-4f42-a7a7-e2566daacb46 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:40:04.965878 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:04.965878 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:04.965878 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:04.965878 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:04.965878 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:04.965878 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:04.965878 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:04.965878 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:04.965878 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:04.965878 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:04.965878 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:04.965878 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:40:04.965878 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:40:04.965878 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:40:04.965878 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:40:04.967160 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:40:04.967160 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:40:04.967160 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:40:04.967160 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:40:04.967160 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:04.967160 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:04.967160 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:04.967160 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:04.967160 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:04.967160 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:04.967160 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:04.967160 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:04.967160 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:04.967160 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:04.967160 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:04.967160 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:04.968197 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:04.968197 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:04.968197 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:04.968197 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:04.968197 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:40:04.968197 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:40:04.968197 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:40:04.968197 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:40:04.968197 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:40:04.968197 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:40:04.968197 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:40:04.968197 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:04.969771 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 385/782] 10.222.0.27 () {56 vars in 1096 bytes} [Sun Sep 11 00:40:04 2022] GET /identity/v3/roles/admin => generated 83 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:04.979980 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-778e0f02-d60a-4f42-a7a7-e2566daacb46 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:05.004536 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d34cb38f-397e-4205-b477-92c1cece53bb None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:05.072286 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee1e1f2a-1c87-4fc2-8037-9c29c84928b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:05.073209 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee1e1f2a-1c87-4fc2-8037-9c29c84928b3 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:05.073675 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee1e1f2a-1c87-4fc2-8037-9c29c84928b3 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:05.074108 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee1e1f2a-1c87-4fc2-8037-9c29c84928b3 None admin] PATH_INFO: `/v3/services` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:05.074978 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee1e1f2a-1c87-4fc2-8037-9c29c84928b3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:05.082026 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee1e1f2a-1c87-4fc2-8037-9c29c84928b3 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:05.089399 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 398/783] 10.222.0.27 () {56 vars in 1087 bytes} [Sun Sep 11 00:40:04 2022] GET /identity/v3/services => generated 772 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:05.126602 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184849f3-b812-4170-b595-d824222e6e47 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:05.126969 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184849f3-b812-4170-b595-d824222e6e47 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:05.127192 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184849f3-b812-4170-b595-d824222e6e47 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:05.128831 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 399/784] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:40:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 11 00:40:05.135452 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f126e5-769a-4665-8094-5e717ef2dcf6 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:05.135742 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f126e5-769a-4665-8094-5e717ef2dcf6 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:05.136008 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f126e5-769a-4665-8094-5e717ef2dcf6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:05.159205 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-47f126e5-769a-4665-8094-5e717ef2dcf6 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:40:05.160683 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-47f126e5-769a-4665-8094-5e717ef2dcf6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:40:05.166955 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-778e0f02-d60a-4f42-a7a7-e2566daacb46 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:05.210232 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:05.218432 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c4d945-2120-49bb-bf82-532bfc00c368 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:05.219083 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c4d945-2120-49bb-bf82-532bfc00c368 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:05.219446 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c4d945-2120-49bb-bf82-532bfc00c368 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:05.219717 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c4d945-2120-49bb-bf82-532bfc00c368 None admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:05.220385 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28c4d945-2120-49bb-bf82-532bfc00c368 None admin] RBAC: Adding query filter params (name=admin) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:05.220679 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28c4d945-2120-49bb-bf82-532bfc00c368 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:05.222082 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28c4d945-2120-49bb-bf82-532bfc00c368 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:05.232508 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 386/785] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:40:04 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:05.238126 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c4d945-2120-49bb-bf82-532bfc00c368 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:05.254891 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 400/786] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:40:05 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 11 00:40:05.300856 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:05.374567 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c4d945-2120-49bb-bf82-532bfc00c368 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:05.414846 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d41f8c9-8813-4122-8735-e32436a064ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:05.415480 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d41f8c9-8813-4122-8735-e32436a064ae None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:05.415752 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d41f8c9-8813-4122-8735-e32436a064ae None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:05.416018 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d41f8c9-8813-4122-8735-e32436a064ae None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:05.427131 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d41f8c9-8813-4122-8735-e32436a064ae None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:05.429240 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d41f8c9-8813-4122-8735-e32436a064ae None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:05.444532 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-1d41f8c9-8813-4122-8735-e32436a064ae None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:40:05.444532 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:05.444532 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:05.444532 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:05.444532 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:05.444532 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:05.444532 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:05.444532 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:05.444532 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:05.444532 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:05.444532 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:05.444532 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:05.444532 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:40:05.444532 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:40:05.444532 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:40:05.444532 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:40:05.445848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:05.445848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:05.445848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:05.445848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:05.445848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:05.445848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:05.445848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:05.445848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:05.445848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:05.445848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:05.445848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:05.445848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:05.445848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:05.445848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:05.445848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:05.445848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:05.445848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:40:05.446732 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:40:05.446732 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:40:05.446732 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:05.446732 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 387/787] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:40:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:40:05.451677 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f126e5-769a-4665-8094-5e717ef2dcf6 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:05.567978 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f126e5-769a-4665-8094-5e717ef2dcf6 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:05.587213 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d41f8c9-8813-4122-8735-e32436a064ae None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:05.625784 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a114b9-2b1b-421c-b054-7bd5817f18de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:05.626312 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a114b9-2b1b-421c-b054-7bd5817f18de None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:05.626580 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a114b9-2b1b-421c-b054-7bd5817f18de None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:05.626861 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a114b9-2b1b-421c-b054-7bd5817f18de None admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:05.627507 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49a114b9-2b1b-421c-b054-7bd5817f18de None admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:05.627798 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49a114b9-2b1b-421c-b054-7bd5817f18de None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:05.629174 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49a114b9-2b1b-421c-b054-7bd5817f18de None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:05.654164 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 401/788] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:40:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:05.659961 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a114b9-2b1b-421c-b054-7bd5817f18de None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:05.720523 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:05.744849 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:05.750810 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d41f8c9-8813-4122-8735-e32436a064ae None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:05.809349 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b8c32e1-7d95-4cfb-b052-ff44be064894 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:05.810233 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8c32e1-7d95-4cfb-b052-ff44be064894 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:05.811840 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8c32e1-7d95-4cfb-b052-ff44be064894 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:05.812264 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8c32e1-7d95-4cfb-b052-ff44be064894 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:05.835257 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b8c32e1-7d95-4cfb-b052-ff44be064894 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:40:05.841593 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-0b8c32e1-7d95-4cfb-b052-ff44be064894 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:40:05.841593 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:05.841593 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:05.841593 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:05.841593 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:05.841593 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:05.841593 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:05.841593 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:05.841593 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:05.841593 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:05.841593 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:05.841593 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:05.841593 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:40:05.841593 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:40:05.841593 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:40:05.841593 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:40:05.842716 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:40:05.842716 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:40:05.842716 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:40:05.842716 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:40:05.842716 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:05.842716 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:05.842716 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:05.842716 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:05.842716 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:05.842716 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:05.842716 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:05.842716 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:05.842716 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:05.842716 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:05.842716 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:05.842716 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:05.843780 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:05.843780 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:05.843780 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:05.843780 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:05.843780 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:40:05.843780 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:40:05.843780 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:40:05.843780 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:40:05.843780 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:40:05.843780 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:40:05.843780 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:40:05.843780 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:05.844429 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 388/789] 10.222.0.27 () {56 vars in 1096 bytes} [Sun Sep 11 00:40:05 2022] GET /identity/v3/roles/admin => generated 83 bytes in 257 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:40:05.846726 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a114b9-2b1b-421c-b054-7bd5817f18de None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:05.849767 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b8c32e1-7d95-4cfb-b052-ff44be064894 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:05.915876 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9fe1cfd-3c42-48c9-b8ca-50da47c8be2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:05.916959 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9fe1cfd-3c42-48c9-b8ca-50da47c8be2b None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:05.918300 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9fe1cfd-3c42-48c9-b8ca-50da47c8be2b None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:05.918778 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9fe1cfd-3c42-48c9-b8ca-50da47c8be2b None admin] PATH_INFO: `/v3/users/swift` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:05.932594 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9fe1cfd-3c42-48c9-b8ca-50da47c8be2b None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:05.936384 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9fe1cfd-3c42-48c9-b8ca-50da47c8be2b None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:05.960266 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-b9fe1cfd-3c42-48c9-b8ca-50da47c8be2b None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 11 00:40:05.960266 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:05.960266 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:05.960266 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:05.960266 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:05.960266 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:05.960266 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:05.960266 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:05.960266 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:05.960266 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:05.960266 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:05.960266 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:05.960266 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 11 00:40:05.960266 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 11 00:40:05.960266 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 11 00:40:05.960266 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 11 00:40:05.961390 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:05.961390 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:05.961390 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 11 00:40:05.961390 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:40:05.961390 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 11 00:40:05.961390 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:40:05.961390 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:05.961390 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:05.961390 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:05.961390 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:05.961390 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:05.961390 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:05.961390 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:05.961390 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:05.961390 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:05.961390 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:05.961390 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:05.961390 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:05.962365 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:05.962365 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:05.962365 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 11 00:40:05.962365 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 11 00:40:05.962365 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 11 00:40:05.962365 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 11 00:40:05.962365 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 11 00:40:05.962365 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 11 00:40:05.962365 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 11 00:40:05.962365 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:05.965156 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 402/790] 10.222.0.27 () {56 vars in 1096 bytes} [Sun Sep 11 00:40:05 2022] GET /identity/v3/users/swift => generated 83 bytes in 305 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:40:05.970837 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9fe1cfd-3c42-48c9-b8ca-50da47c8be2b None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:06.058330 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b8c32e1-7d95-4cfb-b052-ff44be064894 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:06.127187 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:06.140616 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9fe1cfd-3c42-48c9-b8ca-50da47c8be2b None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:06.155213 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f89c9f32-ec22-4479-8e0d-8b77710d6233 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:06.155828 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89c9f32-ec22-4479-8e0d-8b77710d6233 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:06.156100 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89c9f32-ec22-4479-8e0d-8b77710d6233 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:06.156366 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89c9f32-ec22-4479-8e0d-8b77710d6233 None admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:06.157031 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f89c9f32-ec22-4479-8e0d-8b77710d6233 None admin] RBAC: Adding query filter params (name=admin) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:06.157328 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f89c9f32-ec22-4479-8e0d-8b77710d6233 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:06.158710 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f89c9f32-ec22-4479-8e0d-8b77710d6233 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:06.178979 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 389/791] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:40:05 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 329 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:40:06.184619 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f89c9f32-ec22-4479-8e0d-8b77710d6233 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:06.187906 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e24f851a-fd14-4bc4-9b9c-b79f6c6821c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:06.188646 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24f851a-fd14-4bc4-9b9c-b79f6c6821c6 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:06.188925 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24f851a-fd14-4bc4-9b9c-b79f6c6821c6 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:06.189194 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24f851a-fd14-4bc4-9b9c-b79f6c6821c6 None admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:06.189940 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e24f851a-fd14-4bc4-9b9c-b79f6c6821c6 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:06.190245 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e24f851a-fd14-4bc4-9b9c-b79f6c6821c6 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:06.191704 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e24f851a-fd14-4bc4-9b9c-b79f6c6821c6 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:06.207618 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:06.228090 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 403/792] 10.222.0.27 () {56 vars in 1135 bytes} [Sun Sep 11 00:40:05 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:06.232898 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e24f851a-fd14-4bc4-9b9c-b79f6c6821c6 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:06.323028 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f89c9f32-ec22-4479-8e0d-8b77710d6233 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:06.393439 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2394cfea-6b5d-482a-b442-528534dd1424 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:06.394344 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2394cfea-6b5d-482a-b442-528534dd1424 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:06.394627 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2394cfea-6b5d-482a-b442-528534dd1424 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:06.395082 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2394cfea-6b5d-482a-b442-528534dd1424 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:06.398815 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e24f851a-fd14-4bc4-9b9c-b79f6c6821c6 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:06.408025 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2394cfea-6b5d-482a-b442-528534dd1424 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:06.409873 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2394cfea-6b5d-482a-b442-528534dd1424 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:06.431170 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-2394cfea-6b5d-482a-b442-528534dd1424 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:40:06.431170 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:06.431170 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:06.431170 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:06.431170 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:06.431170 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:06.431170 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:06.431170 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:06.431170 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:06.431170 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:06.431170 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:06.431170 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:06.431170 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:40:06.431170 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:40:06.431170 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:40:06.431170 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:40:06.432127 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:06.432127 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:06.432127 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:06.432127 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:06.432127 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:06.432127 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:06.432127 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:06.432127 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:06.432127 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:06.432127 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:06.432127 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:06.432127 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:06.432127 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:06.432127 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:06.432127 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:06.432127 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:06.432127 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:40:06.433024 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:40:06.433024 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:40:06.433024 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:06.433024 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 390/793] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:40:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 248 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:06.438818 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af6a06f-8ba2-4196-afab-4f1c0686b65b None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:06.438818 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af6a06f-8ba2-4196-afab-4f1c0686b65b None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:06.438818 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af6a06f-8ba2-4196-afab-4f1c0686b65b None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:06.438818 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 391/794] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:40:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:40:06.439903 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9af6a06f-8ba2-4196-afab-4f1c0686b65b None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:06.448070 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-204392bb-047b-4cef-833f-2c248e79c6f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:06.448708 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-204392bb-047b-4cef-833f-2c248e79c6f2 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:06.448985 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-204392bb-047b-4cef-833f-2c248e79c6f2 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:06.449252 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-204392bb-047b-4cef-833f-2c248e79c6f2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:06.461690 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-204392bb-047b-4cef-833f-2c248e79c6f2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:06.463164 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-204392bb-047b-4cef-833f-2c248e79c6f2 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:06.473769 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:06.482334 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-204392bb-047b-4cef-833f-2c248e79c6f2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:40:06.482334 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:06.482334 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:06.482334 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:06.482334 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:06.482334 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:06.482334 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:06.482334 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:06.482334 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:06.482334 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:06.482334 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:06.482334 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:06.482334 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:40:06.482334 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:40:06.482334 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:40:06.482334 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:40:06.483519 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:06.483519 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:06.483519 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:06.483519 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:06.483519 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:06.483519 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:06.483519 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:06.483519 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:06.483519 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:06.483519 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:06.483519 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:06.483519 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:06.483519 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:06.483519 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:06.483519 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:06.483519 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:06.483519 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:40:06.484627 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:40:06.484627 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:40:06.484627 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:06.491830 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 404/795] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:40:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 259 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:40:06.493114 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3357401d-2924-4f07-ae27-b629ac00fc42 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:06.493396 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3357401d-2924-4f07-ae27-b629ac00fc42 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:06.493669 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3357401d-2924-4f07-ae27-b629ac00fc42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:06.522169 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-3357401d-2924-4f07-ae27-b629ac00fc42 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:40:06.524007 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-3357401d-2924-4f07-ae27-b629ac00fc42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:40:06.540939 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:06.583357 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9af6a06f-8ba2-4196-afab-4f1c0686b65b None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:06.609913 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 405/796] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:40:06 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:40:06.611195 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3357401d-2924-4f07-ae27-b629ac00fc42 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:06.621469 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9c3f07-4298-4055-b3d5-0ff69d0807ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:06.622239 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9c3f07-4298-4055-b3d5-0ff69d0807ba None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:06.622606 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9c3f07-4298-4055-b3d5-0ff69d0807ba None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:06.622997 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9c3f07-4298-4055-b3d5-0ff69d0807ba None admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:06.623828 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb9c3f07-4298-4055-b3d5-0ff69d0807ba None admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:06.624222 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb9c3f07-4298-4055-b3d5-0ff69d0807ba None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:06.625886 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb9c3f07-4298-4055-b3d5-0ff69d0807ba None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:06.643605 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 392/797] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:40:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:06.644771 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9c3f07-4298-4055-b3d5-0ff69d0807ba None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:06.723289 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3357401d-2924-4f07-ae27-b629ac00fc42 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:06.762773 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:06.780579 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30f9af07-c612-4a3a-b4d9-a90670b2557d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:06.781184 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f9af07-c612-4a3a-b4d9-a90670b2557d None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:06.781460 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f9af07-c612-4a3a-b4d9-a90670b2557d None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:06.781725 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f9af07-c612-4a3a-b4d9-a90670b2557d None admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:06.782356 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30f9af07-c612-4a3a-b4d9-a90670b2557d None admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:06.782648 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30f9af07-c612-4a3a-b4d9-a90670b2557d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:06.784650 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9c3f07-4298-4055-b3d5-0ff69d0807ba None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:06.787875 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30f9af07-c612-4a3a-b4d9-a90670b2557d None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:06.801720 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 406/798] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:40:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:06.807581 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30f9af07-c612-4a3a-b4d9-a90670b2557d None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:06.849718 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:06.862707 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98cc2210-0956-475a-88f4-466eb2e70a7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:06.863375 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98cc2210-0956-475a-88f4-466eb2e70a7d None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:06.863651 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98cc2210-0956-475a-88f4-466eb2e70a7d None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:06.863922 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98cc2210-0956-475a-88f4-466eb2e70a7d None admin] PATH_INFO: `/v3/users/nova` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:06.875365 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98cc2210-0956-475a-88f4-466eb2e70a7d None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:06.876823 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98cc2210-0956-475a-88f4-466eb2e70a7d None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:06.893060 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-98cc2210-0956-475a-88f4-466eb2e70a7d None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 11 00:40:06.893060 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:06.893060 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:06.893060 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:06.893060 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:06.893060 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:06.893060 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:06.893060 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:06.893060 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:06.893060 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:06.893060 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:06.893060 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:06.893060 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 11 00:40:06.893060 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 11 00:40:06.893060 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 11 00:40:06.893060 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 11 00:40:06.894036 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:06.894036 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:06.894036 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 11 00:40:06.894036 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:40:06.894036 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 11 00:40:06.894036 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:40:06.894036 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:06.894036 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:06.894036 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:06.894036 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:06.894036 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:06.894036 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:06.894036 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:06.894036 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:06.894036 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:06.894036 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:06.894036 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:06.894036 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:06.895037 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:06.895037 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:06.895037 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 11 00:40:06.895037 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 11 00:40:06.895037 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 11 00:40:06.895037 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 11 00:40:06.895037 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 11 00:40:06.895037 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 11 00:40:06.895037 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 11 00:40:06.895037 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:06.895037 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 393/799] 10.222.0.27 () {56 vars in 1093 bytes} [Sun Sep 11 00:40:06 2022] GET /identity/v3/users/nova => generated 82 bytes in 250 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:06.899791 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98cc2210-0956-475a-88f4-466eb2e70a7d None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:07.000111 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30f9af07-c612-4a3a-b4d9-a90670b2557d None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:07.023246 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98cc2210-0956-475a-88f4-466eb2e70a7d None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:07.041888 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cd285c9-d596-449b-add4-231c502cbecf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:07.043120 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd285c9-d596-449b-add4-231c502cbecf None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:07.043692 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd285c9-d596-449b-add4-231c502cbecf None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:07.044566 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd285c9-d596-449b-add4-231c502cbecf None admin] PATH_INFO: `/v3/projects/service` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:07.053224 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cd285c9-d596-449b-add4-231c502cbecf None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:07.055879 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cd285c9-d596-449b-add4-231c502cbecf None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:07.063332 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa255a0d-d179-4a3b-9415-1249d030d715 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:07.064018 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa255a0d-d179-4a3b-9415-1249d030d715 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:07.064365 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa255a0d-d179-4a3b-9415-1249d030d715 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:07.064744 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa255a0d-d179-4a3b-9415-1249d030d715 None admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:07.065535 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa255a0d-d179-4a3b-9415-1249d030d715 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:07.065939 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa255a0d-d179-4a3b-9415-1249d030d715 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:07.067421 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa255a0d-d179-4a3b-9415-1249d030d715 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:07.071714 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-2cd285c9-d596-449b-add4-231c502cbecf None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:40:07.071714 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:07.071714 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:07.071714 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:07.071714 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:07.071714 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:07.071714 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:07.071714 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:07.071714 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:07.071714 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:07.071714 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:07.071714 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:07.071714 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 11 00:40:07.071714 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 11 00:40:07.071714 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 11 00:40:07.072611 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 11 00:40:07.072611 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:07.072611 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:07.072611 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:07.072611 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:07.072611 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:07.072611 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:07.072611 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:07.072611 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:07.072611 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:07.072611 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:07.072611 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:07.072611 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:07.072611 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:07.072611 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:07.072611 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:07.072611 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:07.073537 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 11 00:40:07.073537 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 11 00:40:07.073537 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 11 00:40:07.073537 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 11 00:40:07.073537 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 11 00:40:07.073537 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 11 00:40:07.073537 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:40:07.073537 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:07.076201 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 407/800] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:40:06 2022] GET /identity/v3/projects/service => generated 88 bytes in 269 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:07.077337 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cd285c9-d596-449b-add4-231c502cbecf None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:07.091198 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 394/801] 10.222.0.27 () {56 vars in 1133 bytes} [Sun Sep 11 00:40:06 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:40:07.092466 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa255a0d-d179-4a3b-9415-1249d030d715 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:07.140396 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:07.152284 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:07.193233 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cd285c9-d596-449b-add4-231c502cbecf None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:07.210508 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa255a0d-d179-4a3b-9415-1249d030d715 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:07.230068 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bffe32a-aefb-495f-9a0e-22a56b19fb2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:07.230720 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bffe32a-aefb-495f-9a0e-22a56b19fb2f None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:07.231133 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bffe32a-aefb-495f-9a0e-22a56b19fb2f None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:07.231410 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bffe32a-aefb-495f-9a0e-22a56b19fb2f None admin] PATH_INFO: `/v3/roles/service` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:07.244015 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bffe32a-aefb-495f-9a0e-22a56b19fb2f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:40:07.249728 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-9bffe32a-aefb-495f-9a0e-22a56b19fb2f None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 11 00:40:07.249728 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:07.249728 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:07.249728 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:07.249728 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:07.249728 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:07.249728 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:07.249728 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:07.249728 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:07.249728 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:07.249728 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:07.249728 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:07.249728 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:40:07.249728 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:40:07.249728 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:40:07.249728 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:40:07.250871 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:40:07.250871 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:40:07.250871 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:40:07.250871 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:40:07.250871 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:07.250871 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:07.250871 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:07.250871 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:07.250871 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:07.250871 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:07.250871 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:07.250871 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:07.250871 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:07.250871 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:07.250871 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:07.250871 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:07.251758 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:07.251758 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:07.251758 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:07.251758 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:07.251758 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:40:07.251758 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:40:07.251758 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:40:07.251758 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:40:07.251758 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:40:07.251758 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:40:07.251758 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 11 00:40:07.251758 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:07.252418 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 408/802] 10.222.0.27 () {56 vars in 1102 bytes} [Sun Sep 11 00:40:07 2022] GET /identity/v3/roles/service => generated 85 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:07.253232 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bffe32a-aefb-495f-9a0e-22a56b19fb2f None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:07.367219 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bffe32a-aefb-495f-9a0e-22a56b19fb2f None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:07.379286 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:07.380320 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bf8fc19-43f4-49ec-b244-3eb9a2fbbf4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:07.381047 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf8fc19-43f4-49ec-b244-3eb9a2fbbf4d None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:07.381424 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf8fc19-43f4-49ec-b244-3eb9a2fbbf4d None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:07.381700 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf8fc19-43f4-49ec-b244-3eb9a2fbbf4d None admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:07.383119 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bf8fc19-43f4-49ec-b244-3eb9a2fbbf4d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:07.383671 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bf8fc19-43f4-49ec-b244-3eb9a2fbbf4d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:07.385847 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bf8fc19-43f4-49ec-b244-3eb9a2fbbf4d None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:07.404971 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 395/803] 10.222.0.27 () {56 vars in 1148 bytes} [Sun Sep 11 00:40:07 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 313 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:07.406458 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bf8fc19-43f4-49ec-b244-3eb9a2fbbf4d None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:07.415777 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226b29b3-9330-4405-bb1a-45031bd44a20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:07.416712 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226b29b3-9330-4405-bb1a-45031bd44a20 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:07.417215 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226b29b3-9330-4405-bb1a-45031bd44a20 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:07.418195 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226b29b3-9330-4405-bb1a-45031bd44a20 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:07.507859 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-226b29b3-9330-4405-bb1a-45031bd44a20 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:07.510020 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-226b29b3-9330-4405-bb1a-45031bd44a20 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:07.542474 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bf8fc19-43f4-49ec-b244-3eb9a2fbbf4d None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:07.554479 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:07.580216 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a8507a-24d6-4a7c-b14f-d06ad58b7942 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:07.580815 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a8507a-24d6-4a7c-b14f-d06ad58b7942 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:07.581088 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a8507a-24d6-4a7c-b14f-d06ad58b7942 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:07.581356 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a8507a-24d6-4a7c-b14f-d06ad58b7942 None admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:07.582012 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1a8507a-24d6-4a7c-b14f-d06ad58b7942 None admin] RBAC: Adding query filter params (name=service) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:07.582308 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1a8507a-24d6-4a7c-b14f-d06ad58b7942 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:07.583699 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1a8507a-24d6-4a7c-b14f-d06ad58b7942 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:07.592456 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 396/804] 10.222.0.27 () {56 vars in 1103 bytes} [Sun Sep 11 00:40:07 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:07.593491 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a8507a-24d6-4a7c-b14f-d06ad58b7942 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:07.623289 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 409/805] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:40:07 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 11 00:40:07.625020 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b7d87b1-3f5e-4d28-b685-d6bea9cdee3b None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:07.625370 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b7d87b1-3f5e-4d28-b685-d6bea9cdee3b None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:07.625734 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b7d87b1-3f5e-4d28-b685-d6bea9cdee3b None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:07.631809 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 410/806] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:40:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:40:07.631809 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b7d87b1-3f5e-4d28-b685-d6bea9cdee3b None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:07.644144 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:07.713526 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a8507a-24d6-4a7c-b14f-d06ad58b7942 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:07.745292 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b7d87b1-3f5e-4d28-b685-d6bea9cdee3b None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:07.750266 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-023430fd-4d67-4b1e-8ad1-319a7a06ee7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:07.750950 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023430fd-4d67-4b1e-8ad1-319a7a06ee7d None admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:07.751224 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023430fd-4d67-4b1e-8ad1-319a7a06ee7d None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:07.751485 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023430fd-4d67-4b1e-8ad1-319a7a06ee7d None admin] PATH_INFO: `/v3/projects/b504a5058bc6474c8cc077dfeec16799/users/bcb73dec73464a3abc4b774abb4f416f/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:07.753392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-023430fd-4d67-4b1e-8ad1-319a7a06ee7d None admin] RBAC: Authorizing `identity:create_grant(project_id=b504a5058bc6474c8cc077dfeec16799, user_id=bcb73dec73464a3abc4b774abb4f416f, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:07.754833 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-023430fd-4d67-4b1e-8ad1-319a7a06ee7d None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:07.767965 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 397/807] 10.222.0.27 () {58 vars in 1439 bytes} [Sun Sep 11 00:40:07 2022] PUT /identity/v3/projects/b504a5058bc6474c8cc077dfeec16799/users/bcb73dec73464a3abc4b774abb4f416f/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:40:07.768948 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-023430fd-4d67-4b1e-8ad1-319a7a06ee7d None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:07.781790 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae3f69a4-e480-41b5-9583-93d845f898cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:07.782443 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3f69a4-e480-41b5-9583-93d845f898cc None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:07.782727 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3f69a4-e480-41b5-9583-93d845f898cc None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:07.783030 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3f69a4-e480-41b5-9583-93d845f898cc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:07.794968 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae3f69a4-e480-41b5-9583-93d845f898cc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:07.796457 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae3f69a4-e480-41b5-9583-93d845f898cc None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:07.813609 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:07.814183 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-ae3f69a4-e480-41b5-9583-93d845f898cc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:40:07.814183 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:07.814183 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:07.814183 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:07.814183 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:07.814183 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:07.814183 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:07.814183 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:07.814183 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:07.814183 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:07.814183 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:07.814183 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:07.814183 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:40:07.814183 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:40:07.814183 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:40:07.814183 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:40:07.815127 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:07.815127 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:07.815127 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:07.815127 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:07.815127 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:07.815127 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:07.815127 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:07.815127 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:07.815127 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:07.815127 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:07.815127 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:07.815127 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:07.815127 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:07.815127 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:07.815127 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:07.815127 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:07.815127 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:40:07.816016 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:40:07.816016 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:40:07.816016 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:07.816016 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 411/808] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:40:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:07.816901 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3dc4d4-7b68-4b36-9972-044f41e03e05 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:07.816901 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3dc4d4-7b68-4b36-9972-044f41e03e05 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:07.816901 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3dc4d4-7b68-4b36-9972-044f41e03e05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:07.839210 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-5a3dc4d4-7b68-4b36-9972-044f41e03e05 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:40:07.840474 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-5a3dc4d4-7b68-4b36-9972-044f41e03e05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:40:07.899169 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-023430fd-4d67-4b1e-8ad1-319a7a06ee7d None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:07.905808 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:07.920777 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 412/809] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:40:07 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:40:07.922289 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf2b4442-137d-4b03-b4dc-eb06e274a295 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:07.922691 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf2b4442-137d-4b03-b4dc-eb06e274a295 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:07.923083 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf2b4442-137d-4b03-b4dc-eb06e274a295 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:07.924613 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 413/810] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:40:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:40:07.925286 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf2b4442-137d-4b03-b4dc-eb06e274a295 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:07.949695 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3175807-ad54-409b-bcc1-fa09e4a41d5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:07.950056 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3175807-ad54-409b-bcc1-fa09e4a41d5b None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:07.950239 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3175807-ad54-409b-bcc1-fa09e4a41d5b None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:07.950496 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3175807-ad54-409b-bcc1-fa09e4a41d5b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:07.963328 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3175807-ad54-409b-bcc1-fa09e4a41d5b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:07.964781 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3175807-ad54-409b-bcc1-fa09e4a41d5b None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:07.981125 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-f3175807-ad54-409b-bcc1-fa09e4a41d5b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:40:07.981125 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:07.981125 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:07.981125 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:07.981125 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:07.981125 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:07.981125 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:07.981125 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:07.981125 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:07.981125 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:07.981125 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:07.981125 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:07.981125 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:40:07.981125 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:40:07.981125 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:40:07.981125 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:40:07.982106 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:07.982106 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:07.982106 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:07.982106 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:07.982106 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:07.982106 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:07.982106 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:07.982106 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:07.982106 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:07.982106 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:07.982106 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:07.982106 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:07.982106 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:07.982106 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:07.982106 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:07.982106 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:07.982106 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:40:07.983459 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:40:07.983459 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:40:07.983459 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:07.983459 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 398/811] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:40:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:07.988493 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d6e1e56-d6b5-4657-9e69-f13f26ae828c None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:07.988493 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d6e1e56-d6b5-4657-9e69-f13f26ae828c None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:07.988493 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d6e1e56-d6b5-4657-9e69-f13f26ae828c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:08.013060 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-6d6e1e56-d6b5-4657-9e69-f13f26ae828c None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:40:08.015297 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-6d6e1e56-d6b5-4657-9e69-f13f26ae828c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:40:08.065091 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf2b4442-137d-4b03-b4dc-eb06e274a295 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:08.103520 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 399/812] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:40:07 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:40:08.104952 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d6e1e56-d6b5-4657-9e69-f13f26ae828c None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:08.113878 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9b4bf48-354a-4b1b-9b82-e1a6cfd18d15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:08.114843 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b4bf48-354a-4b1b-9b82-e1a6cfd18d15 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:08.115075 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b4bf48-354a-4b1b-9b82-e1a6cfd18d15 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:08.115349 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b4bf48-354a-4b1b-9b82-e1a6cfd18d15 None admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:08.116665 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9b4bf48-354a-4b1b-9b82-e1a6cfd18d15 None admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:08.116963 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9b4bf48-354a-4b1b-9b82-e1a6cfd18d15 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:08.118548 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9b4bf48-354a-4b1b-9b82-e1a6cfd18d15 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:08.126300 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:08.161013 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 414/813] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:40:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:08.167445 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9b4bf48-354a-4b1b-9b82-e1a6cfd18d15 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:08.252679 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:08.264034 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d6e1e56-d6b5-4657-9e69-f13f26ae828c None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:08.306041 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9b4bf48-354a-4b1b-9b82-e1a6cfd18d15 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:08.311066 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cc6632c-8328-4eb5-aa8b-f18f0cf679ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:08.311757 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc6632c-8328-4eb5-aa8b-f18f0cf679ff None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:08.312106 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc6632c-8328-4eb5-aa8b-f18f0cf679ff None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:08.312447 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc6632c-8328-4eb5-aa8b-f18f0cf679ff None admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:08.313174 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cc6632c-8328-4eb5-aa8b-f18f0cf679ff None admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:08.313546 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cc6632c-8328-4eb5-aa8b-f18f0cf679ff None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:08.315643 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cc6632c-8328-4eb5-aa8b-f18f0cf679ff None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:08.335947 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 400/814] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:40:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:08.336813 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cc6632c-8328-4eb5-aa8b-f18f0cf679ff None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:08.351368 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6cf020e-1721-40bb-bdb8-16bb1fae4cdc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:08.352012 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6cf020e-1721-40bb-bdb8-16bb1fae4cdc None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:08.352290 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6cf020e-1721-40bb-bdb8-16bb1fae4cdc None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:08.352557 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6cf020e-1721-40bb-bdb8-16bb1fae4cdc None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:08.365048 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6cf020e-1721-40bb-bdb8-16bb1fae4cdc None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:08.367907 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6cf020e-1721-40bb-bdb8-16bb1fae4cdc None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:08.384111 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-d6cf020e-1721-40bb-bdb8-16bb1fae4cdc None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 11 00:40:08.384111 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:08.384111 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:08.384111 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:08.384111 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:08.384111 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:08.384111 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:08.384111 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:08.384111 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:08.384111 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:08.384111 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:08.384111 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:08.384111 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 11 00:40:08.384111 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 11 00:40:08.384111 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 11 00:40:08.384111 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 11 00:40:08.385077 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:08.385077 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:08.385077 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 11 00:40:08.385077 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:40:08.385077 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 11 00:40:08.385077 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:40:08.385077 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:08.385077 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:08.385077 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:08.385077 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:08.385077 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:08.385077 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:08.385077 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:08.385077 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:08.385077 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:08.385077 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:08.385077 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:08.385077 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:08.386039 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:08.386039 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:08.386039 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 11 00:40:08.386039 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 11 00:40:08.386039 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 11 00:40:08.386039 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 11 00:40:08.386039 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 11 00:40:08.386039 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 11 00:40:08.386039 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 11 00:40:08.386039 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:08.389326 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 415/815] 10.222.0.27 () {56 vars in 1117 bytes} [Sun Sep 11 00:40:08 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:40:08.390314 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6cf020e-1721-40bb-bdb8-16bb1fae4cdc None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:08.434042 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:08.461586 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cc6632c-8328-4eb5-aa8b-f18f0cf679ff None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:08.497966 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-006b0437-979d-4e38-8723-b108eb561680 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:08.498593 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-006b0437-979d-4e38-8723-b108eb561680 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:08.498900 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-006b0437-979d-4e38-8723-b108eb561680 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:08.499171 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-006b0437-979d-4e38-8723-b108eb561680 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:08.499753 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-006b0437-979d-4e38-8723-b108eb561680 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:08.501346 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-006b0437-979d-4e38-8723-b108eb561680 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:08.509773 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6cf020e-1721-40bb-bdb8-16bb1fae4cdc None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:08.511376 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-006b0437-979d-4e38-8723-b108eb561680 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 11 00:40:08.511376 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:08.511376 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 11 00:40:08.511376 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 11 00:40:08.511376 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 11 00:40:08.511376 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 11 00:40:08.511376 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 11 00:40:08.511376 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 11 00:40:08.511376 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 11 00:40:08.511376 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:08.511376 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 11 00:40:08.511376 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:08.511376 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:08.511376 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:08.511376 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:08.511376 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:08.511376 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:08.512350 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:08.512350 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:08.512350 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:08.512350 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:08.512350 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:08.512350 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:08.512350 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 11 00:40:08.512350 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 11 00:40:08.512350 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 11 00:40:08.512350 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 11 00:40:08.512350 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:08.512350 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:08.512350 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:08.512350 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:08.512350 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:08.512350 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:08.513199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:08.513199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:08.513199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:08.513199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:08.513199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:08.513199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:08.513199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:08.513199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:08.513199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:08.513199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:08.513199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 11 00:40:08.513199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 11 00:40:08.513199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 11 00:40:08.513199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:08.513199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 401/816] 10.222.0.27 () {56 vars in 1129 bytes} [Sun Sep 11 00:40:08 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:08.514586 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-006b0437-979d-4e38-8723-b108eb561680 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:08.528206 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:08.551685 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f7339b2-df08-4f80-b990-9510d58f02f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:08.552346 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7339b2-df08-4f80-b990-9510d58f02f7 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:08.552623 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7339b2-df08-4f80-b990-9510d58f02f7 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:08.552890 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7339b2-df08-4f80-b990-9510d58f02f7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:08.560334 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f7339b2-df08-4f80-b990-9510d58f02f7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:08.561775 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f7339b2-df08-4f80-b990-9510d58f02f7 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:08.573327 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-3f7339b2-df08-4f80-b990-9510d58f02f7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:40:08.573327 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:08.573327 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:08.573327 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:08.573327 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:08.573327 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:08.573327 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:08.573327 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:08.573327 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:08.573327 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:08.573327 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:08.573327 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:08.573327 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 11 00:40:08.573327 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 11 00:40:08.573327 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 11 00:40:08.574220 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 11 00:40:08.574220 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:08.574220 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:08.574220 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:08.574220 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:08.574220 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:08.574220 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:08.574220 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:08.574220 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:08.574220 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:08.574220 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:08.574220 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:08.574220 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:08.574220 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:08.574220 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:08.574220 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:08.574220 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:08.575160 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 11 00:40:08.575160 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 11 00:40:08.575160 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 11 00:40:08.575160 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 11 00:40:08.575160 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 11 00:40:08.575160 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 11 00:40:08.575160 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:40:08.575160 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:08.575160 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 416/817] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:40:08 2022] GET /identity/v3/projects/service => generated 88 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:08.575649 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f7339b2-df08-4f80-b990-9510d58f02f7 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:08.619193 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-006b0437-979d-4e38-8723-b108eb561680 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:08.653311 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b371401d-3dbd-47a9-a9c7-9550877804b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:08.653914 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b371401d-3dbd-47a9-a9c7-9550877804b8 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:08.654188 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b371401d-3dbd-47a9-a9c7-9550877804b8 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:08.654527 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b371401d-3dbd-47a9-a9c7-9550877804b8 None admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:08.655217 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b371401d-3dbd-47a9-a9c7-9550877804b8 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:08.655519 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b371401d-3dbd-47a9-a9c7-9550877804b8 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:08.656959 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b371401d-3dbd-47a9-a9c7-9550877804b8 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:08.666062 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:08.675497 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 402/818] 10.222.0.27 () {56 vars in 1149 bytes} [Sun Sep 11 00:40:08 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:08.676296 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b371401d-3dbd-47a9-a9c7-9550877804b8 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:08.695455 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f7339b2-df08-4f80-b990-9510d58f02f7 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:08.739839 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-434dff27-54b0-4365-8c9c-0bf7e5667f45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:08.740459 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434dff27-54b0-4365-8c9c-0bf7e5667f45 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:08.741401 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434dff27-54b0-4365-8c9c-0bf7e5667f45 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:08.741890 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434dff27-54b0-4365-8c9c-0bf7e5667f45 None admin] PATH_INFO: `/v3/services/network` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:08.742998 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-434dff27-54b0-4365-8c9c-0bf7e5667f45 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:08.744383 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-434dff27-54b0-4365-8c9c-0bf7e5667f45 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:08.753585 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-434dff27-54b0-4365-8c9c-0bf7e5667f45 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 11 00:40:08.753585 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:08.753585 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 11 00:40:08.753585 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 11 00:40:08.753585 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 11 00:40:08.753585 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 11 00:40:08.753585 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 11 00:40:08.753585 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 11 00:40:08.753585 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 11 00:40:08.753585 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:08.753585 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 11 00:40:08.753585 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:08.753585 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:08.753585 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:08.753585 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:08.753585 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:08.753585 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:08.754678 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:08.754678 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:08.754678 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:08.754678 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:08.754678 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:08.754678 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:08.754678 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 11 00:40:08.754678 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 11 00:40:08.754678 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 11 00:40:08.754678 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 11 00:40:08.754678 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:08.754678 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:08.754678 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:08.754678 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:08.754678 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:08.754678 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:08.755581 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:08.755581 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:08.755581 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:08.755581 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:08.755581 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:08.755581 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:08.755581 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:08.755581 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:08.755581 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:08.755581 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:08.755581 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 11 00:40:08.755581 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 11 00:40:08.755581 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 11 00:40:08.755581 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:08.756375 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 417/819] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:40:08 2022] GET /identity/v3/services/network => generated 88 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:08.756972 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-434dff27-54b0-4365-8c9c-0bf7e5667f45 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:08.790932 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b371401d-3dbd-47a9-a9c7-9550877804b8 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:08.798359 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:08.827017 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b981f8b-93ff-43a3-99fb-d4f9beb8d6c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:08.827589 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b981f8b-93ff-43a3-99fb-d4f9beb8d6c4 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:08.827877 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b981f8b-93ff-43a3-99fb-d4f9beb8d6c4 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:08.828150 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b981f8b-93ff-43a3-99fb-d4f9beb8d6c4 None admin] PATH_INFO: `/v3/services` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:08.828777 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b981f8b-93ff-43a3-99fb-d4f9beb8d6c4 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:08.829077 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b981f8b-93ff-43a3-99fb-d4f9beb8d6c4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:08.830441 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b981f8b-93ff-43a3-99fb-d4f9beb8d6c4 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:08.838969 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 403/820] 10.222.0.27 () {56 vars in 1124 bytes} [Sun Sep 11 00:40:08 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:08.838969 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b981f8b-93ff-43a3-99fb-d4f9beb8d6c4 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:08.874467 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-434dff27-54b0-4365-8c9c-0bf7e5667f45 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:08.897716 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:08.912372 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3ac369-186f-4554-8f7c-0c1ac71a7eec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:08.913868 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3ac369-186f-4554-8f7c-0c1ac71a7eec None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:08.914146 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3ac369-186f-4554-8f7c-0c1ac71a7eec None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:08.914442 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3ac369-186f-4554-8f7c-0c1ac71a7eec None admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:08.915125 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff3ac369-186f-4554-8f7c-0c1ac71a7eec None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:08.915659 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff3ac369-186f-4554-8f7c-0c1ac71a7eec None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:08.916827 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff3ac369-186f-4554-8f7c-0c1ac71a7eec None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:08.929344 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 418/821] 10.222.0.27 () {56 vars in 1148 bytes} [Sun Sep 11 00:40:08 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:08.929979 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3ac369-186f-4554-8f7c-0c1ac71a7eec None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:08.947024 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b981f8b-93ff-43a3-99fb-d4f9beb8d6c4 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:08.983520 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-994eca84-5771-4a88-83c6-a7862f46f432 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:08.984145 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994eca84-5771-4a88-83c6-a7862f46f432 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:08.984418 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994eca84-5771-4a88-83c6-a7862f46f432 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:08.984687 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994eca84-5771-4a88-83c6-a7862f46f432 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:08.998250 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-994eca84-5771-4a88-83c6-a7862f46f432 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:08.999736 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-994eca84-5771-4a88-83c6-a7862f46f432 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:09.014665 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-994eca84-5771-4a88-83c6-a7862f46f432 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:40:09.014665 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:09.014665 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:09.014665 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:09.014665 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:09.014665 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:09.014665 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:09.014665 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:09.014665 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:09.014665 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:09.014665 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:09.014665 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:09.014665 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:40:09.014665 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:40:09.014665 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:40:09.014665 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:40:09.015822 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:09.015822 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:09.015822 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:09.015822 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:09.015822 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:09.015822 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:09.015822 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:09.015822 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:09.015822 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:09.015822 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:09.015822 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:09.015822 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:09.015822 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:09.015822 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:09.015822 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:09.015822 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:09.015822 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:40:09.016748 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:40:09.016748 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:40:09.016748 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:09.016748 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 404/822] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:40:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:09.016973 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-994eca84-5771-4a88-83c6-a7862f46f432 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:09.034362 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:09.041616 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3ac369-186f-4554-8f7c-0c1ac71a7eec None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:09.086694 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00d1385b-cae2-4bbb-967b-be24ac3c5671 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:09.087273 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d1385b-cae2-4bbb-967b-be24ac3c5671 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:09.087549 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d1385b-cae2-4bbb-967b-be24ac3c5671 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:09.087822 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d1385b-cae2-4bbb-967b-be24ac3c5671 None admin] PATH_INFO: `/v3/services` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:09.088454 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00d1385b-cae2-4bbb-967b-be24ac3c5671 None admin] RBAC: Adding query filter params (name=network) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:09.088758 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00d1385b-cae2-4bbb-967b-be24ac3c5671 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:09.090141 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00d1385b-cae2-4bbb-967b-be24ac3c5671 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:09.094878 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 419/823] 10.222.0.27 () {56 vars in 1112 bytes} [Sun Sep 11 00:40:08 2022] GET /identity/v3/services?name=network => generated 124 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:09.099686 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00d1385b-cae2-4bbb-967b-be24ac3c5671 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:09.133703 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-994eca84-5771-4a88-83c6-a7862f46f432 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:09.146540 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:09.169633 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bc9bf83-e8ea-47d3-86e5-51d18cef935c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:09.170229 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc9bf83-e8ea-47d3-86e5-51d18cef935c None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:09.170553 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc9bf83-e8ea-47d3-86e5-51d18cef935c None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:09.170853 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc9bf83-e8ea-47d3-86e5-51d18cef935c None admin] PATH_INFO: `/v3/services` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:09.171473 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bc9bf83-e8ea-47d3-86e5-51d18cef935c None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:09.171772 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bc9bf83-e8ea-47d3-86e5-51d18cef935c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:09.173140 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bc9bf83-e8ea-47d3-86e5-51d18cef935c None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:09.183429 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 405/824] 10.222.0.27 () {56 vars in 1124 bytes} [Sun Sep 11 00:40:09 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:09.184340 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bc9bf83-e8ea-47d3-86e5-51d18cef935c None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:09.211292 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00d1385b-cae2-4bbb-967b-be24ac3c5671 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:09.245389 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0deadb20-c3c6-4ae3-a582-f19c6cc317b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:09.245989 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0deadb20-c3c6-4ae3-a582-f19c6cc317b2 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:09.246262 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0deadb20-c3c6-4ae3-a582-f19c6cc317b2 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:09.246528 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0deadb20-c3c6-4ae3-a582-f19c6cc317b2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:09.247276 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0deadb20-c3c6-4ae3-a582-f19c6cc317b2 None admin] RBAC: Adding query filter params (role.id=7d9213c1187c4b5182bf8132cf23ea19, scope.project.id=b504a5058bc6474c8cc077dfeec16799, user.id=835def0e6c6d4365a2767051509eb0d6) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:09.247579 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0deadb20-c3c6-4ae3-a582-f19c6cc317b2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:09.249058 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0deadb20-c3c6-4ae3-a582-f19c6cc317b2 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:09.254765 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 420/825] 10.222.0.27 () {56 vars in 1374 bytes} [Sun Sep 11 00:40:09 2022] GET /identity/v3/role_assignments?user.id=835def0e6c6d4365a2767051509eb0d6&scope.project.id=b504a5058bc6474c8cc077dfeec16799&role.id=7d9213c1187c4b5182bf8132cf23ea19 => generated 608 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:09.255558 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0deadb20-c3c6-4ae3-a582-f19c6cc317b2 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:09.294523 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:09.317941 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bc9bf83-e8ea-47d3-86e5-51d18cef935c None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:09.358773 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0bba7d-481b-418a-af9b-a1621e1d0ec8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:09.359706 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0bba7d-481b-418a-af9b-a1621e1d0ec8 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:09.360301 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0bba7d-481b-418a-af9b-a1621e1d0ec8 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:09.360754 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0bba7d-481b-418a-af9b-a1621e1d0ec8 None admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:09.361598 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d0bba7d-481b-418a-af9b-a1621e1d0ec8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:09.362135 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d0bba7d-481b-418a-af9b-a1621e1d0ec8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:09.364058 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d0bba7d-481b-418a-af9b-a1621e1d0ec8 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:09.378329 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 406/826] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:40:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:09.379675 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0bba7d-481b-418a-af9b-a1621e1d0ec8 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:09.390232 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0deadb20-c3c6-4ae3-a582-f19c6cc317b2 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:09.431681 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:09.436683 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c9a7f6-1c22-4019-a4f8-f1cf58209d5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:09.437665 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c9a7f6-1c22-4019-a4f8-f1cf58209d5c None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:09.439259 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c9a7f6-1c22-4019-a4f8-f1cf58209d5c None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:09.439734 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c9a7f6-1c22-4019-a4f8-f1cf58209d5c None admin] PATH_INFO: `/v3/services` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:09.440564 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c9a7f6-1c22-4019-a4f8-f1cf58209d5c None admin] RBAC: Adding query filter params (type=network) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:09.441623 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c9a7f6-1c22-4019-a4f8-f1cf58209d5c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:09.444363 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c9a7f6-1c22-4019-a4f8-f1cf58209d5c None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:09.450944 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 421/827] 10.222.0.27 () {56 vars in 1112 bytes} [Sun Sep 11 00:40:09 2022] GET /identity/v3/services?type=network => generated 350 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:09.452300 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c9a7f6-1c22-4019-a4f8-f1cf58209d5c None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:09.510826 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0bba7d-481b-418a-af9b-a1621e1d0ec8 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:09.550807 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1da8ecd0-107b-4297-b52e-f90489bcb2bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:09.551417 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da8ecd0-107b-4297-b52e-f90489bcb2bd None admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:09.552233 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da8ecd0-107b-4297-b52e-f90489bcb2bd None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:09.552233 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da8ecd0-107b-4297-b52e-f90489bcb2bd None admin] PATH_INFO: `/v3/endpoints` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:09.552874 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1da8ecd0-107b-4297-b52e-f90489bcb2bd None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:09.555679 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1da8ecd0-107b-4297-b52e-f90489bcb2bd None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:09.575165 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:09.590676 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 407/828] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:40:09 2022] POST /identity/v3/endpoints => generated 346 bytes in 211 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:40:09.594835 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c9a7f6-1c22-4019-a4f8-f1cf58209d5c None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:09.595982 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1da8ecd0-107b-4297-b52e-f90489bcb2bd None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:09.655476 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8108378d-44c9-4253-8b2c-8a8dd172d01e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:09.656048 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8108378d-44c9-4253-8b2c-8a8dd172d01e None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:09.656328 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8108378d-44c9-4253-8b2c-8a8dd172d01e None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:09.656597 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8108378d-44c9-4253-8b2c-8a8dd172d01e None admin] PATH_INFO: `/v3/projects/service` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:09.673367 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8108378d-44c9-4253-8b2c-8a8dd172d01e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:09.675584 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8108378d-44c9-4253-8b2c-8a8dd172d01e None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:09.701231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-8108378d-44c9-4253-8b2c-8a8dd172d01e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:40:09.701231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:09.701231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:09.701231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:09.701231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:09.701231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:09.701231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:09.701231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:09.701231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:09.701231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:09.701231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:09.701231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:09.701231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 11 00:40:09.701231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 11 00:40:09.701231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 11 00:40:09.702227 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 11 00:40:09.702227 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:09.702227 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:09.702227 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:09.702227 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:09.702227 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:09.702227 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:09.702227 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:09.702227 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:09.702227 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:09.702227 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:09.702227 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:09.702227 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:09.702227 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:09.702227 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:09.702227 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:09.702227 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:09.703298 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 11 00:40:09.703298 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 11 00:40:09.703298 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 11 00:40:09.703298 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 11 00:40:09.703298 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 11 00:40:09.703298 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 11 00:40:09.703298 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:40:09.703298 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:09.703725 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 422/829] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:40:09 2022] GET /identity/v3/projects/service => generated 88 bytes in 252 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:09.709759 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8108378d-44c9-4253-8b2c-8a8dd172d01e None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:09.836189 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:09.837115 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8108378d-44c9-4253-8b2c-8a8dd172d01e None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:09.855775 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1da8ecd0-107b-4297-b52e-f90489bcb2bd None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:09.879910 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb1c7e3-5ba5-4b99-b2c5-c7b3ccdbd666 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:09.880904 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb1c7e3-5ba5-4b99-b2c5-c7b3ccdbd666 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:09.881901 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb1c7e3-5ba5-4b99-b2c5-c7b3ccdbd666 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:09.882659 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb1c7e3-5ba5-4b99-b2c5-c7b3ccdbd666 None admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:09.883690 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb1c7e3-5ba5-4b99-b2c5-c7b3ccdbd666 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:09.884370 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb1c7e3-5ba5-4b99-b2c5-c7b3ccdbd666 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:09.886672 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb1c7e3-5ba5-4b99-b2c5-c7b3ccdbd666 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:09.896859 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b60feceb-4b01-4f63-90a2-aec96ef7dea4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:09.897469 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60feceb-4b01-4f63-90a2-aec96ef7dea4 None admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:09.897931 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60feceb-4b01-4f63-90a2-aec96ef7dea4 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:09.898214 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60feceb-4b01-4f63-90a2-aec96ef7dea4 None admin] PATH_INFO: `/v3/endpoints` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:09.898929 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b60feceb-4b01-4f63-90a2-aec96ef7dea4 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:09.900420 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b60feceb-4b01-4f63-90a2-aec96ef7dea4 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:09.903161 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 423/830] 10.222.0.27 () {56 vars in 1148 bytes} [Sun Sep 11 00:40:09 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:09.909629 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb1c7e3-5ba5-4b99-b2c5-c7b3ccdbd666 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:09.916090 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:09.934936 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 408/831] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:40:09 2022] POST /identity/v3/endpoints => generated 337 bytes in 340 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:40:10.035445 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb1c7e3-5ba5-4b99-b2c5-c7b3ccdbd666 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:10.085860 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a407bc4-66b1-454f-a95f-7c6089a809df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:10.086549 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a407bc4-66b1-454f-a95f-7c6089a809df None admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:10.086857 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a407bc4-66b1-454f-a95f-7c6089a809df None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:10.087132 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a407bc4-66b1-454f-a95f-7c6089a809df None admin] PATH_INFO: `/v3/projects/b504a5058bc6474c8cc077dfeec16799/users/9fc4274836844ae4ad5d16d74c36f671/roles/0390f926e7ef4715ae18697033bde90e` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:10.097545 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65760c1-8720-413f-ba3b-6b8f7fcec506 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:10.097851 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65760c1-8720-413f-ba3b-6b8f7fcec506 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:10.098093 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65760c1-8720-413f-ba3b-6b8f7fcec506 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:10.100365 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 409/832] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:40:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:40:10.105432 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a407bc4-66b1-454f-a95f-7c6089a809df None admin] RBAC: Authorizing `identity:create_grant(project_id=b504a5058bc6474c8cc077dfeec16799, user_id=9fc4274836844ae4ad5d16d74c36f671, role_id=0390f926e7ef4715ae18697033bde90e)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:10.106665 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81223650-015a-4f50-948b-cb05e595e946 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:10.106974 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81223650-015a-4f50-948b-cb05e595e946 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:10.107082 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a407bc4-66b1-454f-a95f-7c6089a809df None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:10.107278 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81223650-015a-4f50-948b-cb05e595e946 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:10.121378 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 424/833] 10.222.0.27 () {58 vars in 1439 bytes} [Sun Sep 11 00:40:09 2022] PUT /identity/v3/projects/b504a5058bc6474c8cc077dfeec16799/users/9fc4274836844ae4ad5d16d74c36f671/roles/0390f926e7ef4715ae18697033bde90e => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:40:10.139829 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-81223650-015a-4f50-948b-cb05e595e946 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:40:10.141028 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-81223650-015a-4f50-948b-cb05e595e946 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:40:10.227213 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 410/834] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:40:10 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:40:10.509690 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a407bc4-66b1-454f-a95f-7c6089a809df None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:10.561375 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:10.612905 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a407bc4-66b1-454f-a95f-7c6089a809df None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:10.647752 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-331eddc3-90c8-4ad0-8816-4465619bbe62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:10.648391 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331eddc3-90c8-4ad0-8816-4465619bbe62 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:10.648667 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331eddc3-90c8-4ad0-8816-4465619bbe62 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:10.648939 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331eddc3-90c8-4ad0-8816-4465619bbe62 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:10.659358 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-331eddc3-90c8-4ad0-8816-4465619bbe62 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:40:10.665217 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-331eddc3-90c8-4ad0-8816-4465619bbe62 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:40:10.665217 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:10.665217 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:10.665217 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:10.665217 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:10.665217 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:10.665217 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:10.665217 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:10.665217 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:10.665217 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:10.665217 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:10.665217 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:10.665217 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:40:10.665217 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:40:10.665217 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:40:10.665217 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:40:10.666372 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:40:10.666372 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:40:10.666372 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:40:10.666372 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:40:10.666372 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:10.666372 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:10.666372 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:10.666372 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:10.666372 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:10.666372 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:10.666372 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:10.666372 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:10.666372 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:10.666372 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:10.666372 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:10.666372 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:10.667624 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:10.667624 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:10.667624 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:10.667624 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:10.667624 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:40:10.667624 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:40:10.667624 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:40:10.667624 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:40:10.667624 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:40:10.667624 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:40:10.667624 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:40:10.667624 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:10.667624 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 425/835] 10.222.0.27 () {56 vars in 1096 bytes} [Sun Sep 11 00:40:10 2022] GET /identity/v3/roles/admin => generated 83 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:10.672145 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81223650-015a-4f50-948b-cb05e595e946 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:10.758870 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:10.783965 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81223650-015a-4f50-948b-cb05e595e946 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:10.821662 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68f4c981-a4ae-4e0e-b6c8-6870d9b10abd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:10.822567 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f4c981-a4ae-4e0e-b6c8-6870d9b10abd None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:10.823103 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f4c981-a4ae-4e0e-b6c8-6870d9b10abd None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:10.823558 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f4c981-a4ae-4e0e-b6c8-6870d9b10abd None admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:10.824403 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68f4c981-a4ae-4e0e-b6c8-6870d9b10abd None admin] RBAC: Adding query filter params (name=admin) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:10.824884 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68f4c981-a4ae-4e0e-b6c8-6870d9b10abd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:10.826479 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68f4c981-a4ae-4e0e-b6c8-6870d9b10abd None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:10.835489 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 411/836] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:40:10 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:40:10.842959 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-331eddc3-90c8-4ad0-8816-4465619bbe62 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:10.951551 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-331eddc3-90c8-4ad0-8816-4465619bbe62 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:10.962465 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:10.987189 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51a3fa0e-c3fe-4a1a-ae37-b46449bb563f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:10.987816 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a3fa0e-c3fe-4a1a-ae37-b46449bb563f None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:10.988092 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a3fa0e-c3fe-4a1a-ae37-b46449bb563f None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:10.988359 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a3fa0e-c3fe-4a1a-ae37-b46449bb563f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:10.999550 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51a3fa0e-c3fe-4a1a-ae37-b46449bb563f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:11.001070 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51a3fa0e-c3fe-4a1a-ae37-b46449bb563f None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:11.015868 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-51a3fa0e-c3fe-4a1a-ae37-b46449bb563f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:40:11.015868 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:11.015868 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:11.015868 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:11.015868 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:11.015868 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:11.015868 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:11.015868 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:11.015868 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:11.015868 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:11.015868 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:11.015868 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:11.015868 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:40:11.015868 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:40:11.015868 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:40:11.015868 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:40:11.016841 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:11.016841 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:11.016841 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:11.016841 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:11.016841 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:11.016841 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:11.016841 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:11.016841 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:11.016841 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:11.016841 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:11.016841 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:11.016841 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:11.016841 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:11.016841 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:11.016841 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:11.016841 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:11.016841 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:40:11.018060 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:40:11.018060 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:40:11.018060 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:11.018060 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 426/837] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:40:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:11.023569 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68f4c981-a4ae-4e0e-b6c8-6870d9b10abd None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:11.130586 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68f4c981-a4ae-4e0e-b6c8-6870d9b10abd None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:11.166544 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc59a149-6423-42b8-a5dd-2090e0246477 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:11.167176 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc59a149-6423-42b8-a5dd-2090e0246477 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:11.167447 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc59a149-6423-42b8-a5dd-2090e0246477 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:11.167708 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc59a149-6423-42b8-a5dd-2090e0246477 None admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:11.168383 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc59a149-6423-42b8-a5dd-2090e0246477 None admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:11.168681 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc59a149-6423-42b8-a5dd-2090e0246477 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:11.170034 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc59a149-6423-42b8-a5dd-2090e0246477 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:11.179902 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:11.183542 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 412/838] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:40:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:11.189293 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51a3fa0e-c3fe-4a1a-ae37-b46449bb563f None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:11.292373 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51a3fa0e-c3fe-4a1a-ae37-b46449bb563f None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:11.331089 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e378467b-aba9-401b-b3fb-fc40975ff510 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:11.331721 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e378467b-aba9-401b-b3fb-fc40975ff510 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:11.331998 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e378467b-aba9-401b-b3fb-fc40975ff510 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:11.332271 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e378467b-aba9-401b-b3fb-fc40975ff510 None admin] PATH_INFO: `/v3/users/swift` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:11.340820 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e378467b-aba9-401b-b3fb-fc40975ff510 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:11.342267 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e378467b-aba9-401b-b3fb-fc40975ff510 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:11.353976 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-e378467b-aba9-401b-b3fb-fc40975ff510 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 11 00:40:11.353976 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:11.353976 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:11.353976 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:11.353976 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:11.353976 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:11.353976 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:11.353976 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:11.353976 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:11.353976 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:11.353976 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:11.353976 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:11.353976 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 11 00:40:11.353976 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 11 00:40:11.353976 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 11 00:40:11.353976 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 11 00:40:11.354955 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:11.354955 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:11.354955 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 11 00:40:11.354955 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:40:11.354955 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 11 00:40:11.354955 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:40:11.354955 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:11.354955 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:11.354955 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:11.354955 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:11.354955 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:11.354955 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:11.354955 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:11.354955 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:11.354955 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:11.354955 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:11.354955 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:11.354955 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:11.356237 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:11.356237 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:11.356237 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 11 00:40:11.356237 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 11 00:40:11.356237 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 11 00:40:11.356237 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 11 00:40:11.356237 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 11 00:40:11.356237 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 11 00:40:11.356237 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 11 00:40:11.356237 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:11.356237 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 427/839] 10.222.0.27 () {56 vars in 1096 bytes} [Sun Sep 11 00:40:11 2022] GET /identity/v3/users/swift => generated 83 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:11.360700 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc59a149-6423-42b8-a5dd-2090e0246477 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:11.471690 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc59a149-6423-42b8-a5dd-2090e0246477 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:11.508223 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-521f69e6-55f7-479f-9712-210d3a6b8fb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:11.508821 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521f69e6-55f7-479f-9712-210d3a6b8fb0 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:11.509094 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521f69e6-55f7-479f-9712-210d3a6b8fb0 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:11.509359 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521f69e6-55f7-479f-9712-210d3a6b8fb0 None admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:11.510079 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-521f69e6-55f7-479f-9712-210d3a6b8fb0 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:11.510374 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-521f69e6-55f7-479f-9712-210d3a6b8fb0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:11.511820 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-521f69e6-55f7-479f-9712-210d3a6b8fb0 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:11.529795 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 413/840] 10.222.0.27 () {56 vars in 1135 bytes} [Sun Sep 11 00:40:11 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:11.535572 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e378467b-aba9-401b-b3fb-fc40975ff510 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:11.561091 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:11.639874 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e378467b-aba9-401b-b3fb-fc40975ff510 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:11.678268 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31df70bb-3077-48e8-921d-03e7306918a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:11.678933 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31df70bb-3077-48e8-921d-03e7306918a0 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:11.679213 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31df70bb-3077-48e8-921d-03e7306918a0 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:11.679482 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31df70bb-3077-48e8-921d-03e7306918a0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:11.753234 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31df70bb-3077-48e8-921d-03e7306918a0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:11.754688 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31df70bb-3077-48e8-921d-03e7306918a0 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:11.783141 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:11.812361 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3ca41f-6700-401c-a90c-63479ce2b5cf None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:11.812648 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3ca41f-6700-401c-a90c-63479ce2b5cf None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:11.812923 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3ca41f-6700-401c-a90c-63479ce2b5cf None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:11.814493 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 414/841] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:40:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:40:11.820188 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e681a2e-fef4-4483-a2c6-45206c89b7ac None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:11.820473 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e681a2e-fef4-4483-a2c6-45206c89b7ac None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:11.820748 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e681a2e-fef4-4483-a2c6-45206c89b7ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:11.843210 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-2e681a2e-fef4-4483-a2c6-45206c89b7ac None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:40:11.844416 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-2e681a2e-fef4-4483-a2c6-45206c89b7ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:40:11.857951 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 428/842] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:40:11 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 11 00:40:11.863876 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31df70bb-3077-48e8-921d-03e7306918a0 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:11.917229 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:11.934735 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 415/843] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:40:11 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 11 00:40:11.990915 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31df70bb-3077-48e8-921d-03e7306918a0 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:12.014924 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:12.029120 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-852e6eb5-0d99-49bf-b436-7909ba01c55d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:12.029754 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852e6eb5-0d99-49bf-b436-7909ba01c55d None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:12.030030 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852e6eb5-0d99-49bf-b436-7909ba01c55d None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:12.030366 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852e6eb5-0d99-49bf-b436-7909ba01c55d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:12.041769 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-852e6eb5-0d99-49bf-b436-7909ba01c55d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:12.043240 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-852e6eb5-0d99-49bf-b436-7909ba01c55d None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:12.058952 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-852e6eb5-0d99-49bf-b436-7909ba01c55d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:40:12.058952 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:12.058952 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:12.058952 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:12.058952 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:12.058952 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:12.058952 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:12.058952 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:12.058952 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:12.058952 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:12.058952 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:12.058952 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:12.058952 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:40:12.058952 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:40:12.058952 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:40:12.058952 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:40:12.059914 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:12.059914 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:12.059914 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:12.059914 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:12.059914 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:12.059914 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:12.059914 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:12.059914 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:12.059914 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:12.059914 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:12.059914 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:12.059914 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:12.059914 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:12.059914 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:12.059914 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:12.059914 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:12.059914 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:40:12.060823 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:40:12.060823 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:40:12.060823 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:12.060823 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 429/844] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:40:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:40:12.065783 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e681a2e-fef4-4483-a2c6-45206c89b7ac None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:12.183625 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e681a2e-fef4-4483-a2c6-45206c89b7ac None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:12.221984 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31685e12-b76d-496d-a059-384f188787eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:12.222612 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31685e12-b76d-496d-a059-384f188787eb None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:12.222918 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31685e12-b76d-496d-a059-384f188787eb None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:12.223190 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31685e12-b76d-496d-a059-384f188787eb None admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:12.223852 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31685e12-b76d-496d-a059-384f188787eb None admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:12.224152 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31685e12-b76d-496d-a059-384f188787eb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:12.226097 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31685e12-b76d-496d-a059-384f188787eb None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:12.239064 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 416/845] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:40:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:12.244717 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-852e6eb5-0d99-49bf-b436-7909ba01c55d None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:12.264746 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31685e12-b76d-496d-a059-384f188787eb None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:12.333458 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:12.358080 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-852e6eb5-0d99-49bf-b436-7909ba01c55d None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:12.381553 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31685e12-b76d-496d-a059-384f188787eb None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:12.395094 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f72ba237-8148-423b-84fc-998002e7b5a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:12.395717 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72ba237-8148-423b-84fc-998002e7b5a5 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:12.395988 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72ba237-8148-423b-84fc-998002e7b5a5 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:12.396256 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72ba237-8148-423b-84fc-998002e7b5a5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:12.402894 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f72ba237-8148-423b-84fc-998002e7b5a5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:12.404335 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f72ba237-8148-423b-84fc-998002e7b5a5 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:12.417150 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-f72ba237-8148-423b-84fc-998002e7b5a5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:40:12.417150 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:12.417150 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:12.417150 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:12.417150 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:12.417150 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:12.417150 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:12.417150 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:12.417150 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:12.417150 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:12.417150 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:12.417150 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:12.417150 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 11 00:40:12.417150 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 11 00:40:12.417150 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 11 00:40:12.418071 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 11 00:40:12.418071 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:12.418071 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:12.418071 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:12.418071 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:12.418071 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:12.418071 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:12.418071 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:12.418071 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:12.418071 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:12.418071 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:12.418071 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:12.418071 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:12.418071 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:12.418071 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:12.418071 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:12.418071 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:12.424036 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 11 00:40:12.424036 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 11 00:40:12.424036 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 11 00:40:12.424036 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 11 00:40:12.424036 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 11 00:40:12.424036 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 11 00:40:12.424036 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:40:12.424036 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:12.424036 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 430/846] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:40:12 2022] GET /identity/v3/projects/service => generated 88 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:40:12.424036 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5259865d-486f-4fc9-9f12-d0c13a4012c5 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:12.424036 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5259865d-486f-4fc9-9f12-d0c13a4012c5 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:12.424663 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5259865d-486f-4fc9-9f12-d0c13a4012c5 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:12.424663 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 431/847] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:40:12 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:40:12.424663 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5259865d-486f-4fc9-9f12-d0c13a4012c5 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:12.430120 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f399eeeb-1813-407a-a23e-a0bd54e86fb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:12.430764 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f399eeeb-1813-407a-a23e-a0bd54e86fb9 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:12.431070 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f399eeeb-1813-407a-a23e-a0bd54e86fb9 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:12.431340 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f399eeeb-1813-407a-a23e-a0bd54e86fb9 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:12.431927 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f399eeeb-1813-407a-a23e-a0bd54e86fb9 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:12.433382 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f399eeeb-1813-407a-a23e-a0bd54e86fb9 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:12.444428 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-f399eeeb-1813-407a-a23e-a0bd54e86fb9 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 11 00:40:12.444428 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:12.444428 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 11 00:40:12.444428 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 11 00:40:12.444428 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 11 00:40:12.444428 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 11 00:40:12.444428 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 11 00:40:12.444428 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 11 00:40:12.444428 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 11 00:40:12.444428 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:12.444428 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 11 00:40:12.444428 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:12.444428 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:12.444428 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:12.444428 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:12.444428 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:12.445412 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:12.445412 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:12.445412 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:12.445412 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:12.445412 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:12.445412 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:12.445412 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:12.445412 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 11 00:40:12.445412 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 11 00:40:12.445412 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 11 00:40:12.445412 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 11 00:40:12.445412 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:12.445412 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:12.445412 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:12.445412 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:12.445412 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:12.445412 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:12.446325 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:12.446325 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:12.446325 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:12.446325 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:12.446325 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:12.446325 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:12.446325 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:12.446325 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:12.446325 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:12.446325 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:12.446325 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 11 00:40:12.446325 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 11 00:40:12.446325 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 11 00:40:12.446325 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:12.446325 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 417/848] 10.222.0.27 () {56 vars in 1132 bytes} [Sun Sep 11 00:40:12 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:40:12.448253 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41c2433-4d7b-4a11-81c6-3632677d860a None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:12.448253 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41c2433-4d7b-4a11-81c6-3632677d860a None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:12.448253 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41c2433-4d7b-4a11-81c6-3632677d860a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:12.466609 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-e41c2433-4d7b-4a11-81c6-3632677d860a None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:40:12.467831 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-e41c2433-4d7b-4a11-81c6-3632677d860a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:40:12.469491 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:12.534184 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 418/849] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:40:12 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:40:12.535060 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e41c2433-4d7b-4a11-81c6-3632677d860a None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:12.547501 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5259865d-486f-4fc9-9f12-d0c13a4012c5 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:12.580313 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:12.585378 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47417bca-d508-43ed-84bd-5ed40db2f857 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:12.585992 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47417bca-d508-43ed-84bd-5ed40db2f857 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:12.586271 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47417bca-d508-43ed-84bd-5ed40db2f857 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:12.586535 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47417bca-d508-43ed-84bd-5ed40db2f857 None admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:12.587256 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47417bca-d508-43ed-84bd-5ed40db2f857 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:12.587560 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47417bca-d508-43ed-84bd-5ed40db2f857 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:12.588967 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47417bca-d508-43ed-84bd-5ed40db2f857 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:12.602714 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 432/850] 10.222.0.27 () {56 vars in 1148 bytes} [Sun Sep 11 00:40:12 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:12.604179 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a6caf3a-5e91-47bd-8686-5f1917642262 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:12.604704 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a6caf3a-5e91-47bd-8686-5f1917642262 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:12.605176 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a6caf3a-5e91-47bd-8686-5f1917642262 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:12.606888 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 433/851] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:40:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:40:12.608603 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a6caf3a-5e91-47bd-8686-5f1917642262 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:12.648091 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e41c2433-4d7b-4a11-81c6-3632677d860a None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:12.684288 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e10fdd6-d9c2-456a-94e8-f1ecd9040316 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:12.684982 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e10fdd6-d9c2-456a-94e8-f1ecd9040316 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:12.685260 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e10fdd6-d9c2-456a-94e8-f1ecd9040316 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:12.686181 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e10fdd6-d9c2-456a-94e8-f1ecd9040316 None admin] PATH_INFO: `/v3/services` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:12.686181 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e10fdd6-d9c2-456a-94e8-f1ecd9040316 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:12.686445 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e10fdd6-d9c2-456a-94e8-f1ecd9040316 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:12.687831 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e10fdd6-d9c2-456a-94e8-f1ecd9040316 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:12.697528 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 419/852] 10.222.0.27 () {56 vars in 1126 bytes} [Sun Sep 11 00:40:12 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:12.697528 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e7b531-4cd2-4758-ab77-c9ef792d2c89 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:12.697528 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e7b531-4cd2-4758-ab77-c9ef792d2c89 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:12.697528 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e7b531-4cd2-4758-ab77-c9ef792d2c89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:12.713523 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-60e7b531-4cd2-4758-ab77-c9ef792d2c89 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:40:12.714763 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-60e7b531-4cd2-4758-ab77-c9ef792d2c89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:40:12.716724 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:12.724585 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a6caf3a-5e91-47bd-8686-5f1917642262 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:12.766294 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e913d7be-331d-4cef-a7d3-c94ef1561ead None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:12.767229 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e913d7be-331d-4cef-a7d3-c94ef1561ead None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:12.767689 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e913d7be-331d-4cef-a7d3-c94ef1561ead None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:12.768138 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e913d7be-331d-4cef-a7d3-c94ef1561ead None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:12.769487 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e913d7be-331d-4cef-a7d3-c94ef1561ead None admin] RBAC: Adding query filter params (role.id=7d9213c1187c4b5182bf8132cf23ea19, scope.project.id=b504a5058bc6474c8cc077dfeec16799, user.id=bcb73dec73464a3abc4b774abb4f416f) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:12.770243 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e913d7be-331d-4cef-a7d3-c94ef1561ead None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:12.773173 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e913d7be-331d-4cef-a7d3-c94ef1561ead None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:12.780912 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 434/853] 10.222.0.27 () {56 vars in 1374 bytes} [Sun Sep 11 00:40:12 2022] GET /identity/v3/role_assignments?user.id=bcb73dec73464a3abc4b774abb4f416f&scope.project.id=b504a5058bc6474c8cc077dfeec16799&role.id=7d9213c1187c4b5182bf8132cf23ea19 => generated 608 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:40:12.781882 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e913d7be-331d-4cef-a7d3-c94ef1561ead None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:12.794991 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 420/854] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:40:12 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:40:12.812079 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e7b531-4cd2-4758-ab77-c9ef792d2c89 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:12.851865 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:12.918833 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e913d7be-331d-4cef-a7d3-c94ef1561ead None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:12.926745 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e7b531-4cd2-4758-ab77-c9ef792d2c89 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:12.959105 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a02d584-6532-44c6-9732-e3a3e47755d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:12.960630 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a02d584-6532-44c6-9732-e3a3e47755d8 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:12.961127 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a02d584-6532-44c6-9732-e3a3e47755d8 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:12.961587 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a02d584-6532-44c6-9732-e3a3e47755d8 None admin] PATH_INFO: `/v3/services` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:12.962458 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a02d584-6532-44c6-9732-e3a3e47755d8 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:12.962989 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a02d584-6532-44c6-9732-e3a3e47755d8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:12.964556 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a02d584-6532-44c6-9732-e3a3e47755d8 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:12.967131 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9522d32b-a752-4fb9-b35f-fe6f3f180099 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:12.967985 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9522d32b-a752-4fb9-b35f-fe6f3f180099 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:12.968446 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9522d32b-a752-4fb9-b35f-fe6f3f180099 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:12.968902 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9522d32b-a752-4fb9-b35f-fe6f3f180099 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:12.969668 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9522d32b-a752-4fb9-b35f-fe6f3f180099 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:12.969983 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 435/855] 10.222.0.27 () {56 vars in 1126 bytes} [Sun Sep 11 00:40:12 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:12.971482 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9522d32b-a752-4fb9-b35f-fe6f3f180099 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:12.982818 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-9522d32b-a752-4fb9-b35f-fe6f3f180099 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 11 00:40:12.982818 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:12.982818 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 11 00:40:12.982818 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 11 00:40:12.982818 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 11 00:40:12.982818 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 11 00:40:12.982818 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 11 00:40:12.982818 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 11 00:40:12.982818 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 11 00:40:12.982818 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:12.982818 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 11 00:40:12.982818 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:12.982818 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:12.982818 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:12.982818 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:12.982818 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:12.982818 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:12.983918 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:12.983918 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:12.983918 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:12.983918 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:12.983918 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:12.983918 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:12.983918 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 11 00:40:12.983918 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 11 00:40:12.983918 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 11 00:40:12.983918 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 11 00:40:12.983918 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:12.983918 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:12.983918 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:12.983918 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:12.983918 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:12.983918 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:12.984769 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:12.984769 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:12.984769 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:12.984769 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:12.984769 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:12.984769 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:12.984769 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:12.984769 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:12.984769 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:12.984769 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:12.984769 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 11 00:40:12.984769 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 11 00:40:12.984769 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 11 00:40:12.984769 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:12.985532 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 421/856] 10.222.0.27 () {56 vars in 1114 bytes} [Sun Sep 11 00:40:12 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:12.993401 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a02d584-6532-44c6-9732-e3a3e47755d8 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:13.021658 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:13.073992 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9522d32b-a752-4fb9-b35f-fe6f3f180099 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:13.132888 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a02d584-6532-44c6-9732-e3a3e47755d8 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:13.180738 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-935d2c92-0c24-4211-a7b9-9a7252ae26cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:13.181360 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-935d2c92-0c24-4211-a7b9-9a7252ae26cf None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:13.181639 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-935d2c92-0c24-4211-a7b9-9a7252ae26cf None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:13.181911 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-935d2c92-0c24-4211-a7b9-9a7252ae26cf None admin] PATH_INFO: `/v3/services` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:13.182551 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-935d2c92-0c24-4211-a7b9-9a7252ae26cf None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:13.183029 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-935d2c92-0c24-4211-a7b9-9a7252ae26cf None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:13.184748 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-935d2c92-0c24-4211-a7b9-9a7252ae26cf None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:13.190673 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 436/857] 10.222.0.27 () {56 vars in 1114 bytes} [Sun Sep 11 00:40:12 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:13.197075 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-935d2c92-0c24-4211-a7b9-9a7252ae26cf None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:13.201500 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9522d32b-a752-4fb9-b35f-fe6f3f180099 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:13.211212 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:13.237810 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8031d410-17c0-4d6d-ba29-87e81bb2ca16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:13.238441 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8031d410-17c0-4d6d-ba29-87e81bb2ca16 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:13.238722 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8031d410-17c0-4d6d-ba29-87e81bb2ca16 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:13.239043 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8031d410-17c0-4d6d-ba29-87e81bb2ca16 None admin] PATH_INFO: `/v3/roles/service` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:13.251628 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8031d410-17c0-4d6d-ba29-87e81bb2ca16 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:40:13.256777 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-8031d410-17c0-4d6d-ba29-87e81bb2ca16 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 11 00:40:13.256777 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:13.256777 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:13.256777 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:13.256777 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:13.256777 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:13.256777 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:13.256777 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:13.256777 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:13.256777 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:13.256777 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:13.256777 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:13.256777 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:40:13.256777 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:40:13.256777 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:40:13.256777 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:40:13.257830 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:40:13.257830 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:40:13.257830 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:40:13.257830 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:40:13.257830 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:13.257830 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:13.257830 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:13.257830 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:13.257830 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:13.257830 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:13.257830 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:13.257830 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:13.257830 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:13.257830 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:13.257830 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:13.257830 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:13.258907 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:13.258907 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:13.258907 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:13.258907 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:13.258907 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:40:13.258907 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:40:13.258907 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:40:13.258907 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:40:13.258907 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:40:13.258907 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:40:13.258907 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 11 00:40:13.258907 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:13.258907 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 422/858] 10.222.0.27 () {56 vars in 1102 bytes} [Sun Sep 11 00:40:13 2022] GET /identity/v3/roles/service => generated 85 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:13.265715 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8031d410-17c0-4d6d-ba29-87e81bb2ca16 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:13.300701 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:13.320448 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-935d2c92-0c24-4211-a7b9-9a7252ae26cf None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:13.360295 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da29a186-9835-404a-ba4b-48f9a7c7e342 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:13.360907 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da29a186-9835-404a-ba4b-48f9a7c7e342 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:13.361189 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da29a186-9835-404a-ba4b-48f9a7c7e342 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:13.361519 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da29a186-9835-404a-ba4b-48f9a7c7e342 None admin] PATH_INFO: `/v3/services` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:13.362153 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da29a186-9835-404a-ba4b-48f9a7c7e342 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:13.362457 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da29a186-9835-404a-ba4b-48f9a7c7e342 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:13.363934 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da29a186-9835-404a-ba4b-48f9a7c7e342 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:13.375758 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 437/859] 10.222.0.27 () {56 vars in 1114 bytes} [Sun Sep 11 00:40:13 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:40:13.392974 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8031d410-17c0-4d6d-ba29-87e81bb2ca16 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:13.439194 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245a30ea-5322-4e67-8267-1636d293c4e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:13.439811 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245a30ea-5322-4e67-8267-1636d293c4e6 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:13.440091 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245a30ea-5322-4e67-8267-1636d293c4e6 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:13.440360 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245a30ea-5322-4e67-8267-1636d293c4e6 None admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:13.441038 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-245a30ea-5322-4e67-8267-1636d293c4e6 None admin] RBAC: Adding query filter params (name=service) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:13.441341 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-245a30ea-5322-4e67-8267-1636d293c4e6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:13.442761 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-245a30ea-5322-4e67-8267-1636d293c4e6 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:13.453780 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 423/860] 10.222.0.27 () {56 vars in 1103 bytes} [Sun Sep 11 00:40:13 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:13.464095 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da29a186-9835-404a-ba4b-48f9a7c7e342 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:13.581753 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da29a186-9835-404a-ba4b-48f9a7c7e342 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:13.616703 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38764d7-a0ba-489b-ace3-52c1b80ff2c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:13.617742 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38764d7-a0ba-489b-ace3-52c1b80ff2c5 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:13.618234 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38764d7-a0ba-489b-ace3-52c1b80ff2c5 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:13.618725 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38764d7-a0ba-489b-ace3-52c1b80ff2c5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:13.631753 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b38764d7-a0ba-489b-ace3-52c1b80ff2c5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:13.633485 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b38764d7-a0ba-489b-ace3-52c1b80ff2c5 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:13.648413 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-b38764d7-a0ba-489b-ace3-52c1b80ff2c5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:40:13.648413 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:13.648413 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:13.648413 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:13.648413 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:13.648413 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:13.648413 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:13.648413 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:13.648413 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:13.648413 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:13.648413 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:13.648413 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:13.648413 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:40:13.648413 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:40:13.648413 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:40:13.648413 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:40:13.649495 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:13.649495 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:13.649495 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:13.649495 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:13.649495 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:13.649495 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:13.649495 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:13.649495 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:13.649495 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:13.649495 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:13.649495 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:13.649495 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:13.649495 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:13.649495 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:13.649495 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:13.649495 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:13.649495 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:40:13.650521 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:40:13.650521 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:40:13.650521 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:13.657048 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 438/861] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:40:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:13.658635 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245a30ea-5322-4e67-8267-1636d293c4e6 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:13.687204 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:13.768528 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245a30ea-5322-4e67-8267-1636d293c4e6 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:13.804803 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c22a6f-dd86-4c6d-98c0-53683dd8e1ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:13.805797 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c22a6f-dd86-4c6d-98c0-53683dd8e1ed None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:13.806285 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c22a6f-dd86-4c6d-98c0-53683dd8e1ed None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:13.806762 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c22a6f-dd86-4c6d-98c0-53683dd8e1ed None admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:13.807683 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66c22a6f-dd86-4c6d-98c0-53683dd8e1ed None admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:13.808198 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66c22a6f-dd86-4c6d-98c0-53683dd8e1ed None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:13.809922 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66c22a6f-dd86-4c6d-98c0-53683dd8e1ed None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:13.826377 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 424/862] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:40:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:13.832244 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38764d7-a0ba-489b-ace3-52c1b80ff2c5 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:13.836501 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:13.946902 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38764d7-a0ba-489b-ace3-52c1b80ff2c5 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:13.981945 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d7e61d-e997-4633-9ae2-16fc10e72ab7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:13.982579 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d7e61d-e997-4633-9ae2-16fc10e72ab7 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:13.982885 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d7e61d-e997-4633-9ae2-16fc10e72ab7 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:13.983159 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d7e61d-e997-4633-9ae2-16fc10e72ab7 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:13.991496 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d7e61d-e997-4633-9ae2-16fc10e72ab7 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:13.992950 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d7e61d-e997-4633-9ae2-16fc10e72ab7 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:14.004494 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-43d7e61d-e997-4633-9ae2-16fc10e72ab7 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 11 00:40:14.004494 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:14.004494 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:14.004494 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:14.004494 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:14.004494 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:14.004494 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:14.004494 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:14.004494 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:14.004494 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:14.004494 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:14.004494 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:14.004494 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 11 00:40:14.004494 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 11 00:40:14.004494 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 11 00:40:14.004494 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 11 00:40:14.005454 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:14.005454 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:14.005454 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 11 00:40:14.005454 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:40:14.005454 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 11 00:40:14.005454 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:40:14.005454 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:14.005454 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:14.005454 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:14.005454 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:14.005454 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:14.005454 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:14.005454 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:14.005454 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:14.005454 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:14.005454 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:14.005454 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:14.005454 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:14.007041 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:14.007041 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:14.007041 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 11 00:40:14.007041 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 11 00:40:14.007041 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 11 00:40:14.007041 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 11 00:40:14.007041 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 11 00:40:14.007041 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 11 00:40:14.007041 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 11 00:40:14.007041 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:14.007041 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 439/863] 10.222.0.27 () {56 vars in 1117 bytes} [Sun Sep 11 00:40:13 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:14.013696 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c22a6f-dd86-4c6d-98c0-53683dd8e1ed None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:14.102103 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:14.129551 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c22a6f-dd86-4c6d-98c0-53683dd8e1ed None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:14.168105 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bab6e392-aabe-4bff-9426-2b396799af7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:14.168715 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab6e392-aabe-4bff-9426-2b396799af7d None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:14.168995 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab6e392-aabe-4bff-9426-2b396799af7d None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:14.169262 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab6e392-aabe-4bff-9426-2b396799af7d None admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:14.169994 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bab6e392-aabe-4bff-9426-2b396799af7d None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:14.170294 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bab6e392-aabe-4bff-9426-2b396799af7d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:14.171833 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bab6e392-aabe-4bff-9426-2b396799af7d None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:14.190975 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 425/864] 10.222.0.27 () {56 vars in 1149 bytes} [Sun Sep 11 00:40:14 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:14.199892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d7e61d-e997-4633-9ae2-16fc10e72ab7 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:14.267327 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:14.309439 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d7e61d-e997-4633-9ae2-16fc10e72ab7 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:14.344884 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3407e898-7bd5-41f0-99c7-6e5c7533dd45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:14.345494 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3407e898-7bd5-41f0-99c7-6e5c7533dd45 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:14.345773 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3407e898-7bd5-41f0-99c7-6e5c7533dd45 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:14.346045 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3407e898-7bd5-41f0-99c7-6e5c7533dd45 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:14.418953 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3407e898-7bd5-41f0-99c7-6e5c7533dd45 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:14.420330 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3407e898-7bd5-41f0-99c7-6e5c7533dd45 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:14.482391 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:14.524313 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 440/865] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:40:14 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 325 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 11 00:40:14.531891 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bab6e392-aabe-4bff-9426-2b396799af7d None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:14.640342 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bab6e392-aabe-4bff-9426-2b396799af7d None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:14.680959 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e3c827-6f1f-41d7-bfe5-4f74b96d2402 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:14.681520 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e3c827-6f1f-41d7-bfe5-4f74b96d2402 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:14.681802 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e3c827-6f1f-41d7-bfe5-4f74b96d2402 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:14.682075 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e3c827-6f1f-41d7-bfe5-4f74b96d2402 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:14.690934 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:14.694093 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78e3c827-6f1f-41d7-bfe5-4f74b96d2402 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:14.695767 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78e3c827-6f1f-41d7-bfe5-4f74b96d2402 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:14.712888 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-78e3c827-6f1f-41d7-bfe5-4f74b96d2402 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:40:14.712888 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:14.712888 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:14.712888 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:14.712888 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:14.712888 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:14.712888 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:14.712888 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:14.712888 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:14.712888 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:14.712888 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:14.712888 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:14.712888 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:40:14.712888 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:40:14.712888 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:40:14.712888 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:40:14.713900 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:14.713900 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:14.713900 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:14.713900 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:14.713900 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:14.713900 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:14.713900 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:14.713900 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:14.713900 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:14.713900 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:14.713900 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:14.713900 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:14.713900 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:14.713900 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:14.713900 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:14.713900 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:14.713900 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:40:14.715245 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:40:14.715245 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:40:14.715245 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:14.715245 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 426/866] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:40:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:14.721463 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3407e898-7bd5-41f0-99c7-6e5c7533dd45 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:14.828332 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3407e898-7bd5-41f0-99c7-6e5c7533dd45 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:14.864566 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-724a8528-0e73-4c18-a4b5-4c88dd9bd2be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:14.865765 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-724a8528-0e73-4c18-a4b5-4c88dd9bd2be None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:14.866153 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-724a8528-0e73-4c18-a4b5-4c88dd9bd2be None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:14.866537 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-724a8528-0e73-4c18-a4b5-4c88dd9bd2be None admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:14.867234 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-724a8528-0e73-4c18-a4b5-4c88dd9bd2be None admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:14.867526 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-724a8528-0e73-4c18-a4b5-4c88dd9bd2be None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:14.868909 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-724a8528-0e73-4c18-a4b5-4c88dd9bd2be None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:14.881481 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 441/867] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:40:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:14.887877 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e3c827-6f1f-41d7-bfe5-4f74b96d2402 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:14.996173 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e3c827-6f1f-41d7-bfe5-4f74b96d2402 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:15.035011 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-733223c8-f6a4-4ee1-a8af-1f335dd1bb84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:15.035931 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733223c8-f6a4-4ee1-a8af-1f335dd1bb84 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:15.036210 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733223c8-f6a4-4ee1-a8af-1f335dd1bb84 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:15.036480 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733223c8-f6a4-4ee1-a8af-1f335dd1bb84 None admin] PATH_INFO: `/v3/projects/service` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:15.044683 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-733223c8-f6a4-4ee1-a8af-1f335dd1bb84 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:15.046182 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-733223c8-f6a4-4ee1-a8af-1f335dd1bb84 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:15.059915 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-733223c8-f6a4-4ee1-a8af-1f335dd1bb84 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:40:15.059915 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:15.059915 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:15.059915 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:15.059915 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:15.059915 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:15.059915 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:15.059915 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:15.059915 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:15.059915 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:15.059915 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:15.059915 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:15.059915 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 11 00:40:15.059915 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 11 00:40:15.059915 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 11 00:40:15.060922 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 11 00:40:15.060922 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:15.060922 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:15.060922 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:15.060922 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:15.060922 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:15.060922 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:15.060922 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:15.060922 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:15.060922 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:15.060922 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:15.060922 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:15.060922 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:15.060922 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:15.060922 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:15.060922 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:15.060922 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:15.062261 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 11 00:40:15.062261 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 11 00:40:15.062261 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 11 00:40:15.062261 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 11 00:40:15.062261 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 11 00:40:15.062261 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 11 00:40:15.062261 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:40:15.062261 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:15.062261 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 427/868] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:40:14 2022] GET /identity/v3/projects/service => generated 88 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:15.068894 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-724a8528-0e73-4c18-a4b5-4c88dd9bd2be None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:15.105165 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:15.158838 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6719649-05b6-4932-80bf-b113d29fb37e None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:15.159399 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6719649-05b6-4932-80bf-b113d29fb37e None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:15.159399 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6719649-05b6-4932-80bf-b113d29fb37e None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:15.161397 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 428/869] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:40:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 11 00:40:15.168501 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75bee31-cbe5-4734-aff9-483fa556ad0b None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:15.168781 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75bee31-cbe5-4734-aff9-483fa556ad0b None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:15.169050 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75bee31-cbe5-4734-aff9-483fa556ad0b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:15.192688 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-f75bee31-cbe5-4734-aff9-483fa556ad0b None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:40:15.193206 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-724a8528-0e73-4c18-a4b5-4c88dd9bd2be None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:15.194579 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-f75bee31-cbe5-4734-aff9-483fa556ad0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:40:15.232222 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f8b2302-4885-4a1f-8c09-cac4e355971a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:15.233229 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8b2302-4885-4a1f-8c09-cac4e355971a None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:15.233605 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8b2302-4885-4a1f-8c09-cac4e355971a None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:15.233973 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8b2302-4885-4a1f-8c09-cac4e355971a None admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:15.234899 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f8b2302-4885-4a1f-8c09-cac4e355971a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:15.235400 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f8b2302-4885-4a1f-8c09-cac4e355971a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:15.237132 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f8b2302-4885-4a1f-8c09-cac4e355971a None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:15.251195 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 442/870] 10.222.0.27 () {56 vars in 1148 bytes} [Sun Sep 11 00:40:15 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:15.252225 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70795f8e-92f0-48f5-a957-c79bc561fab8 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:15.252522 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70795f8e-92f0-48f5-a957-c79bc561fab8 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:15.252778 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70795f8e-92f0-48f5-a957-c79bc561fab8 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:15.258712 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 443/871] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:40:15 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:40:15.259534 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70795f8e-92f0-48f5-a957-c79bc561fab8 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:15.259662 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:15.282082 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 429/872] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:40:15 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 11 00:40:15.283708 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ca1383-c5dc-483f-be0c-3f1631705eff None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:15.283995 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ca1383-c5dc-483f-be0c-3f1631705eff None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:15.284263 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ca1383-c5dc-483f-be0c-3f1631705eff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:15.310869 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-d4ca1383-c5dc-483f-be0c-3f1631705eff None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:40:15.312492 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-d4ca1383-c5dc-483f-be0c-3f1631705eff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:40:15.375226 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:15.400357 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70795f8e-92f0-48f5-a957-c79bc561fab8 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:15.401084 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 430/873] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:40:15 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:40:15.439363 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d242a923-06df-4966-bb80-88490cd9fec8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:15.440344 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d242a923-06df-4966-bb80-88490cd9fec8 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:15.440822 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d242a923-06df-4966-bb80-88490cd9fec8 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:15.441287 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d242a923-06df-4966-bb80-88490cd9fec8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:15.442292 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d242a923-06df-4966-bb80-88490cd9fec8 None admin] RBAC: Adding query filter params (role.id=0390f926e7ef4715ae18697033bde90e, scope.project.id=b504a5058bc6474c8cc077dfeec16799, user.id=9fc4274836844ae4ad5d16d74c36f671) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:15.442841 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d242a923-06df-4966-bb80-88490cd9fec8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:15.444578 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d242a923-06df-4966-bb80-88490cd9fec8 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:15.451166 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 444/874] 10.222.0.27 () {56 vars in 1374 bytes} [Sun Sep 11 00:40:15 2022] GET /identity/v3/role_assignments?user.id=9fc4274836844ae4ad5d16d74c36f671&scope.project.id=b504a5058bc6474c8cc077dfeec16799&role.id=0390f926e7ef4715ae18697033bde90e => generated 608 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:15.561176 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ca1383-c5dc-483f-be0c-3f1631705eff None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:15.673971 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ca1383-c5dc-483f-be0c-3f1631705eff None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:15.680777 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d242a923-06df-4966-bb80-88490cd9fec8 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:15.691814 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:15.719703 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-074dc2dc-3a19-44af-afdb-2282fad427a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:15.720334 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074dc2dc-3a19-44af-afdb-2282fad427a2 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:15.720613 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074dc2dc-3a19-44af-afdb-2282fad427a2 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:15.720881 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074dc2dc-3a19-44af-afdb-2282fad427a2 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:15.721696 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-074dc2dc-3a19-44af-afdb-2282fad427a2 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:15.723895 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-074dc2dc-3a19-44af-afdb-2282fad427a2 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:15.736515 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-074dc2dc-3a19-44af-afdb-2282fad427a2 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 11 00:40:15.736515 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:15.736515 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 11 00:40:15.736515 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 11 00:40:15.736515 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 11 00:40:15.736515 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 11 00:40:15.736515 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 11 00:40:15.736515 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 11 00:40:15.736515 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 11 00:40:15.736515 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:15.736515 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 11 00:40:15.736515 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:15.736515 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:15.736515 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:15.736515 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:15.736515 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:15.736515 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:15.737686 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:15.737686 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:15.737686 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:15.737686 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:15.737686 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:15.737686 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:15.737686 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 11 00:40:15.737686 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 11 00:40:15.737686 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 11 00:40:15.737686 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 11 00:40:15.737686 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:15.737686 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:15.737686 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:15.737686 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:15.737686 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:15.737686 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:15.739160 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:15.739160 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:15.739160 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:15.739160 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:15.739160 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:15.739160 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:15.739160 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:15.739160 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:15.739160 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:15.739160 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:15.739160 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 11 00:40:15.739160 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 11 00:40:15.739160 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 11 00:40:15.739160 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:15.739160 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 431/875] 10.222.0.27 () {56 vars in 1126 bytes} [Sun Sep 11 00:40:15 2022] GET /identity/v3/services/object-store => generated 93 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:15.746144 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-074dc2dc-3a19-44af-afdb-2282fad427a2 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:15.811010 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d242a923-06df-4966-bb80-88490cd9fec8 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:15.846609 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9788436e-f9bc-4049-9f97-657240211d40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:15.847171 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9788436e-f9bc-4049-9f97-657240211d40 None admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:15.847447 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9788436e-f9bc-4049-9f97-657240211d40 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:15.847715 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9788436e-f9bc-4049-9f97-657240211d40 None admin] PATH_INFO: `/v3/services` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:15.848369 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9788436e-f9bc-4049-9f97-657240211d40 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:15.850333 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9788436e-f9bc-4049-9f97-657240211d40 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:15.868373 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 445/876] 10.222.0.27 () {60 vars in 1141 bytes} [Sun Sep 11 00:40:15 2022] POST /identity/v3/services => generated 269 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:40:15.869448 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b13307-1a42-4005-8aa6-7500695b879a None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:15.869774 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b13307-1a42-4005-8aa6-7500695b879a None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:15.870051 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b13307-1a42-4005-8aa6-7500695b879a None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:15.871558 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 446/877] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:40:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:40:15.877724 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3efc05ae-742f-45be-976d-24c6848c99ab None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:15.878027 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3efc05ae-742f-45be-976d-24c6848c99ab None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:15.878257 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3efc05ae-742f-45be-976d-24c6848c99ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:15.890137 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-074dc2dc-3a19-44af-afdb-2282fad427a2 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:15.895879 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:15.903521 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-3efc05ae-742f-45be-976d-24c6848c99ab None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:40:15.905524 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-3efc05ae-742f-45be-976d-24c6848c99ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:40:15.942170 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8899fadb-a6c4-4a4e-843d-affa740a989d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:15.942983 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8899fadb-a6c4-4a4e-843d-affa740a989d None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:15.943336 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8899fadb-a6c4-4a4e-843d-affa740a989d None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:15.944524 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8899fadb-a6c4-4a4e-843d-affa740a989d None admin] PATH_INFO: `/v3/services` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:15.945627 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8899fadb-a6c4-4a4e-843d-affa740a989d None admin] RBAC: Adding query filter params (name=object-store) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:15.946054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8899fadb-a6c4-4a4e-843d-affa740a989d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:15.947690 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8899fadb-a6c4-4a4e-843d-affa740a989d None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:15.953869 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 432/878] 10.222.0.27 () {56 vars in 1122 bytes} [Sun Sep 11 00:40:15 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:40:15.959955 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8899fadb-a6c4-4a4e-843d-affa740a989d None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:16.002347 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 447/879] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:40:15 2022] POST /identity/v3/auth/tokens => generated 1271 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:40:16.003919 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:16.089905 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8899fadb-a6c4-4a4e-843d-affa740a989d None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:16.127086 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-036bcb2b-fc1c-4626-a5c6-83b3b2ca9842 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:16.128015 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036bcb2b-fc1c-4626-a5c6-83b3b2ca9842 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:16.128486 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036bcb2b-fc1c-4626-a5c6-83b3b2ca9842 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:16.128950 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036bcb2b-fc1c-4626-a5c6-83b3b2ca9842 None admin] PATH_INFO: `/v3/services` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:16.129757 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-036bcb2b-fc1c-4626-a5c6-83b3b2ca9842 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:16.130255 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-036bcb2b-fc1c-4626-a5c6-83b3b2ca9842 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:16.132492 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-036bcb2b-fc1c-4626-a5c6-83b3b2ca9842 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:16.137849 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 433/880] 10.222.0.27 () {56 vars in 1122 bytes} [Sun Sep 11 00:40:15 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:40:16.297936 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3efc05ae-742f-45be-976d-24c6848c99ab None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:16.402441 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3efc05ae-742f-45be-976d-24c6848c99ab None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:16.438627 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ed7f73-b4af-405a-ade2-f95f9b7ad48b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:16.439643 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ed7f73-b4af-405a-ade2-f95f9b7ad48b None admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:16.440126 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ed7f73-b4af-405a-ade2-f95f9b7ad48b None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:16.440618 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ed7f73-b4af-405a-ade2-f95f9b7ad48b None admin] PATH_INFO: `/v3/services` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:16.441486 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98ed7f73-b4af-405a-ade2-f95f9b7ad48b None admin] RBAC: Authorizing `identity:create_service()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:16.443086 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98ed7f73-b4af-405a-ade2-f95f9b7ad48b None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:16.448399 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:16.464415 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 448/881] 10.222.0.27 () {60 vars in 1141 bytes} [Sun Sep 11 00:40:16 2022] POST /identity/v3/services => generated 251 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:40:17.778536 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a717d6-e2c6-4719-834b-687e7fd766a9 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:17.785020 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a717d6-e2c6-4719-834b-687e7fd766a9 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:17.785020 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a717d6-e2c6-4719-834b-687e7fd766a9 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:17.785020 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 434/882] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:40:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:40:17.786425 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0998c019-f6b8-4232-9621-bd2e20505408 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:17.786911 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0998c019-f6b8-4232-9621-bd2e20505408 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:17.787159 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0998c019-f6b8-4232-9621-bd2e20505408 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:17.809219 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-0998c019-f6b8-4232-9621-bd2e20505408 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:40:17.810390 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-0998c019-f6b8-4232-9621-bd2e20505408 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:40:17.885056 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 449/883] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:40:17 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:40:18.155941 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a717d6-e2c6-4719-834b-687e7fd766a9 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:18.158515 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658a2872-d845-4b8b-a335-f49bbb1b8e70 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:18.159206 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658a2872-d845-4b8b-a335-f49bbb1b8e70 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:18.159480 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658a2872-d845-4b8b-a335-f49bbb1b8e70 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:18.163166 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 450/884] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:40:18 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 11 00:40:18.169177 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ec668c-725e-4d77-a2f6-065e290dab23 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:18.169455 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ec668c-725e-4d77-a2f6-065e290dab23 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:18.170197 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ec668c-725e-4d77-a2f6-065e290dab23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:18.195969 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-76ec668c-725e-4d77-a2f6-065e290dab23 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:40:18.197364 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-76ec668c-725e-4d77-a2f6-065e290dab23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:40:18.265934 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:18.270450 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a717d6-e2c6-4719-834b-687e7fd766a9 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:18.290877 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 451/885] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:40:18 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 11 00:40:18.311725 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96c6a14c-4370-47da-9703-bc0e6d07fe8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:18.312676 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c6a14c-4370-47da-9703-bc0e6d07fe8c None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:18.313118 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c6a14c-4370-47da-9703-bc0e6d07fe8c None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:18.313552 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c6a14c-4370-47da-9703-bc0e6d07fe8c None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:18.327715 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96c6a14c-4370-47da-9703-bc0e6d07fe8c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:40:18.333145 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-96c6a14c-4370-47da-9703-bc0e6d07fe8c None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 11 00:40:18.333145 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:18.333145 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:18.333145 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:18.333145 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:18.333145 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:18.333145 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:18.333145 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:18.333145 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:18.333145 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:18.333145 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:18.333145 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:18.333145 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:40:18.333145 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:40:18.333145 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:40:18.333145 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:40:18.334535 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:40:18.334535 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:40:18.334535 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:40:18.334535 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:40:18.334535 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:18.334535 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:18.334535 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:18.334535 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:18.334535 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:18.334535 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:18.334535 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:18.334535 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:18.334535 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:18.334535 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:18.334535 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:18.334535 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:18.335871 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:18.335871 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:18.335871 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:18.335871 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:18.335871 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:40:18.335871 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:40:18.335871 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:40:18.335871 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:40:18.335871 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:40:18.335871 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:40:18.335871 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 11 00:40:18.335871 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:18.338320 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 435/886] 10.222.0.27 () {56 vars in 1120 bytes} [Sun Sep 11 00:40:18 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:18.344674 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76ec668c-725e-4d77-a2f6-065e290dab23 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:18.400119 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:18.445940 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de1f600-5b6c-4163-b111-d043dc6fe651 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:18.446221 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de1f600-5b6c-4163-b111-d043dc6fe651 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:18.446869 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de1f600-5b6c-4163-b111-d043dc6fe651 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:18.448932 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 436/887] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:40:18 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 11 00:40:18.454499 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b061fad9-8afe-4759-8303-ba98396d452a None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:18.454780 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b061fad9-8afe-4759-8303-ba98396d452a None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:18.455072 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b061fad9-8afe-4759-8303-ba98396d452a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:18.456220 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76ec668c-725e-4d77-a2f6-065e290dab23 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:18.480647 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-b061fad9-8afe-4759-8303-ba98396d452a None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:40:18.481856 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-b061fad9-8afe-4759-8303-ba98396d452a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:40:18.498862 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8dafd30-f83c-4dfc-81ef-c3ce44c6fc35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:18.499481 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8dafd30-f83c-4dfc-81ef-c3ce44c6fc35 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:18.500052 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8dafd30-f83c-4dfc-81ef-c3ce44c6fc35 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:18.500330 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8dafd30-f83c-4dfc-81ef-c3ce44c6fc35 None admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:18.501494 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8dafd30-f83c-4dfc-81ef-c3ce44c6fc35 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:18.501797 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8dafd30-f83c-4dfc-81ef-c3ce44c6fc35 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:18.503589 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8dafd30-f83c-4dfc-81ef-c3ce44c6fc35 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:18.513536 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 452/888] 10.222.0.27 () {56 vars in 1115 bytes} [Sun Sep 11 00:40:18 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:18.519842 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8dafd30-f83c-4dfc-81ef-c3ce44c6fc35 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:18.574955 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 437/889] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:40:18 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:40:18.598096 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b061fad9-8afe-4759-8303-ba98396d452a None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:18.666281 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8dafd30-f83c-4dfc-81ef-c3ce44c6fc35 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:18.684944 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:18.696317 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:18.707592 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b061fad9-8afe-4759-8303-ba98396d452a None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:18.708316 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3395e66-96c5-4df3-982e-24c6242866d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:18.709062 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3395e66-96c5-4df3-982e-24c6242866d0 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:18.709325 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3395e66-96c5-4df3-982e-24c6242866d0 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:18.709592 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3395e66-96c5-4df3-982e-24c6242866d0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:18.720283 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3395e66-96c5-4df3-982e-24c6242866d0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:18.721709 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3395e66-96c5-4df3-982e-24c6242866d0 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:18.736554 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-f3395e66-96c5-4df3-982e-24c6242866d0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:40:18.736554 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:18.736554 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:18.736554 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:18.736554 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:18.736554 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:18.736554 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:18.736554 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:18.736554 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:18.736554 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:18.736554 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:18.736554 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:18.736554 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:40:18.736554 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:40:18.736554 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:40:18.736554 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:40:18.737542 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:18.737542 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:18.737542 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:18.737542 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:18.737542 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:18.737542 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:18.737542 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:18.737542 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:18.737542 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:18.737542 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:18.737542 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:18.737542 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:18.737542 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:18.737542 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:18.737542 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:18.737542 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:18.737542 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:40:18.738464 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:40:18.738464 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:40:18.738464 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:18.738464 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 453/890] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:40:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:18.739275 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c27abc83-7d10-4914-88e2-c3c7e6983632 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:18.739275 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c27abc83-7d10-4914-88e2-c3c7e6983632 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:18.739275 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c27abc83-7d10-4914-88e2-c3c7e6983632 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:18.741240 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 454/891] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:40:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:40:18.745797 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c27abc83-7d10-4914-88e2-c3c7e6983632 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:18.749033 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0abd38fa-c5be-4295-a78a-78003af36a55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:18.749673 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abd38fa-c5be-4295-a78a-78003af36a55 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:18.749954 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abd38fa-c5be-4295-a78a-78003af36a55 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:18.750236 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abd38fa-c5be-4295-a78a-78003af36a55 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:18.750840 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0abd38fa-c5be-4295-a78a-78003af36a55 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:18.752210 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0abd38fa-c5be-4295-a78a-78003af36a55 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:18.762344 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-0abd38fa-c5be-4295-a78a-78003af36a55 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 11 00:40:18.762344 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:18.762344 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 11 00:40:18.762344 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 11 00:40:18.762344 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 11 00:40:18.762344 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 11 00:40:18.762344 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 11 00:40:18.762344 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 11 00:40:18.762344 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 11 00:40:18.762344 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:18.762344 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 11 00:40:18.762344 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:18.762344 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:18.762344 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:18.762344 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:18.762344 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:18.763392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:18.763392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:18.763392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:18.763392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:18.763392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:18.763392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:18.763392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:18.763392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 11 00:40:18.763392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 11 00:40:18.763392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 11 00:40:18.763392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 11 00:40:18.763392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:18.763392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:18.763392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:18.763392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:18.763392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:18.763392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:18.764331 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:18.764331 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:18.764331 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:18.764331 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:18.764331 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:18.764331 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:18.764331 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:18.764331 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:18.764331 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:18.764331 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:18.764331 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 11 00:40:18.764331 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 11 00:40:18.764331 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 11 00:40:18.764331 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:18.764331 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 438/892] 10.222.0.27 () {56 vars in 1132 bytes} [Sun Sep 11 00:40:18 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:40:18.765591 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51c0711-3044-4645-b165-a6d9f23c90ed None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:18.765591 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51c0711-3044-4645-b165-a6d9f23c90ed None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:18.765591 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51c0711-3044-4645-b165-a6d9f23c90ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:18.782971 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-d51c0711-3044-4645-b165-a6d9f23c90ed None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:40:18.784156 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-d51c0711-3044-4645-b165-a6d9f23c90ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:40:18.855582 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 439/893] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:40:18 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:40:18.856539 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d51c0711-3044-4645-b165-a6d9f23c90ed None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:18.868103 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c27abc83-7d10-4914-88e2-c3c7e6983632 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:18.903620 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43646ef2-f001-4432-a713-5dd73ec4ed5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:18.904463 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43646ef2-f001-4432-a713-5dd73ec4ed5f None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:18.904743 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43646ef2-f001-4432-a713-5dd73ec4ed5f None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:18.905014 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43646ef2-f001-4432-a713-5dd73ec4ed5f None admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:18.905651 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43646ef2-f001-4432-a713-5dd73ec4ed5f None admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:18.905976 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43646ef2-f001-4432-a713-5dd73ec4ed5f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:18.907341 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43646ef2-f001-4432-a713-5dd73ec4ed5f None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:18.923033 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 455/894] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:40:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:18.923033 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43646ef2-f001-4432-a713-5dd73ec4ed5f None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:18.946884 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:18.951435 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:18.963717 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d51c0711-3044-4645-b165-a6d9f23c90ed None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:18.998633 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20957515-b395-4927-ba27-437c0dedf003 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:18.999289 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20957515-b395-4927-ba27-437c0dedf003 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:18.999561 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20957515-b395-4927-ba27-437c0dedf003 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:18.999830 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20957515-b395-4927-ba27-437c0dedf003 None admin] PATH_INFO: `/v3/services` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:19.000461 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20957515-b395-4927-ba27-437c0dedf003 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:19.000761 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20957515-b395-4927-ba27-437c0dedf003 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:19.002121 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20957515-b395-4927-ba27-437c0dedf003 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:19.006812 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 440/895] 10.222.0.27 () {56 vars in 1126 bytes} [Sun Sep 11 00:40:18 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:19.007679 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20957515-b395-4927-ba27-437c0dedf003 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:19.035389 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43646ef2-f001-4432-a713-5dd73ec4ed5f None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:19.072862 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d66c796-8c4d-4c7d-a4d0-cc84feb20aa5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:19.073472 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d66c796-8c4d-4c7d-a4d0-cc84feb20aa5 None admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:19.073748 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d66c796-8c4d-4c7d-a4d0-cc84feb20aa5 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:19.074017 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d66c796-8c4d-4c7d-a4d0-cc84feb20aa5 None admin] PATH_INFO: `/v3/services` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:19.074658 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d66c796-8c4d-4c7d-a4d0-cc84feb20aa5 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:19.076056 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d66c796-8c4d-4c7d-a4d0-cc84feb20aa5 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:19.092275 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 456/896] 10.222.0.27 () {60 vars in 1141 bytes} [Sun Sep 11 00:40:18 2022] POST /identity/v3/services => generated 241 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:40:19.096401 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d66c796-8c4d-4c7d-a4d0-cc84feb20aa5 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:19.122522 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20957515-b395-4927-ba27-437c0dedf003 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:19.171547 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a77cdb-dd66-4874-ae4b-f73506873d57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:19.172408 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a77cdb-dd66-4874-ae4b-f73506873d57 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:19.172521 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a77cdb-dd66-4874-ae4b-f73506873d57 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:19.172761 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a77cdb-dd66-4874-ae4b-f73506873d57 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:19.181569 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3a77cdb-dd66-4874-ae4b-f73506873d57 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:19.183294 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3a77cdb-dd66-4874-ae4b-f73506873d57 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:19.196429 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-b3a77cdb-dd66-4874-ae4b-f73506873d57 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 11 00:40:19.196429 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:19.196429 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:19.196429 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:19.196429 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:19.196429 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:19.196429 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:19.196429 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:19.196429 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:19.196429 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:19.196429 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:19.196429 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:19.196429 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 11 00:40:19.196429 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 11 00:40:19.196429 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 11 00:40:19.196429 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 11 00:40:19.197502 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:19.197502 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:19.197502 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 11 00:40:19.197502 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:40:19.197502 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 11 00:40:19.197502 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:40:19.197502 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:19.197502 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:19.197502 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:19.197502 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:19.197502 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:19.197502 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:19.197502 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:19.197502 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:19.197502 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:19.197502 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:19.197502 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:19.197502 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:19.198543 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:19.198543 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:19.198543 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 11 00:40:19.198543 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 11 00:40:19.198543 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 11 00:40:19.198543 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 11 00:40:19.198543 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 11 00:40:19.198543 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 11 00:40:19.198543 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 11 00:40:19.198543 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:19.202809 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 441/897] 10.222.0.27 () {56 vars in 1117 bytes} [Sun Sep 11 00:40:19 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:19.203547 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a77cdb-dd66-4874-ae4b-f73506873d57 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:19.208647 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:19.213175 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:19.235948 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d66c796-8c4d-4c7d-a4d0-cc84feb20aa5 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:19.277533 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa66f7e-8e8d-4293-885a-a014f7406098 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:19.278143 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa66f7e-8e8d-4293-885a-a014f7406098 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:19.278420 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa66f7e-8e8d-4293-885a-a014f7406098 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:19.279400 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa66f7e-8e8d-4293-885a-a014f7406098 None admin] PATH_INFO: `/v3/services` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:19.279528 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fa66f7e-8e8d-4293-885a-a014f7406098 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:19.279856 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fa66f7e-8e8d-4293-885a-a014f7406098 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:19.281234 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fa66f7e-8e8d-4293-885a-a014f7406098 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:19.286478 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 457/898] 10.222.0.27 () {56 vars in 1126 bytes} [Sun Sep 11 00:40:19 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:19.287528 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa66f7e-8e8d-4293-885a-a014f7406098 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:19.323690 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a77cdb-dd66-4874-ae4b-f73506873d57 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:19.362837 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c9d4c0-1eec-4171-922a-aa2b645657a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:19.363468 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c9d4c0-1eec-4171-922a-aa2b645657a1 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:19.363747 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c9d4c0-1eec-4171-922a-aa2b645657a1 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:19.364057 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c9d4c0-1eec-4171-922a-aa2b645657a1 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:19.364595 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58c9d4c0-1eec-4171-922a-aa2b645657a1 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:19.366092 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58c9d4c0-1eec-4171-922a-aa2b645657a1 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:19.375421 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-58c9d4c0-1eec-4171-922a-aa2b645657a1 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 11 00:40:19.375421 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:19.375421 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 11 00:40:19.375421 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 11 00:40:19.375421 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 11 00:40:19.375421 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 11 00:40:19.375421 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 11 00:40:19.375421 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 11 00:40:19.375421 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 11 00:40:19.375421 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:19.375421 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 11 00:40:19.375421 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:19.375421 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:19.375421 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:19.375421 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:19.375421 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:19.375421 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:19.376548 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:19.376548 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:19.376548 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:19.376548 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:19.376548 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:19.376548 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:19.376548 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 11 00:40:19.376548 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 11 00:40:19.376548 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 11 00:40:19.376548 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 11 00:40:19.376548 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:19.376548 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:19.376548 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:19.376548 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:19.376548 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:19.376548 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:19.377392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:19.377392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:19.377392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:19.377392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:19.377392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:19.377392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:19.377392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:19.377392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:19.377392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:19.377392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:19.377392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 11 00:40:19.377392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 11 00:40:19.377392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 11 00:40:19.377392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:19.377392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 442/899] 10.222.0.27 () {56 vars in 1114 bytes} [Sun Sep 11 00:40:19 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:19.378222 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c9d4c0-1eec-4171-922a-aa2b645657a1 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:19.407973 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa66f7e-8e8d-4293-885a-a014f7406098 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:19.446378 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a47febaf-a675-4337-8919-0bd4114666e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:19.447051 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a47febaf-a675-4337-8919-0bd4114666e9 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:19.447330 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a47febaf-a675-4337-8919-0bd4114666e9 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:19.447595 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a47febaf-a675-4337-8919-0bd4114666e9 None admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:19.448330 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a47febaf-a675-4337-8919-0bd4114666e9 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:19.448589 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:19.448706 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a47febaf-a675-4337-8919-0bd4114666e9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:19.450081 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a47febaf-a675-4337-8919-0bd4114666e9 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:19.469096 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 458/900] 10.222.0.27 () {56 vars in 1149 bytes} [Sun Sep 11 00:40:19 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:19.469750 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a47febaf-a675-4337-8919-0bd4114666e9 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:19.481841 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:19.490294 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c9d4c0-1eec-4171-922a-aa2b645657a1 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:19.528598 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c093ff2-aeaa-4a89-b07e-88e08f1bf155 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:19.529216 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c093ff2-aeaa-4a89-b07e-88e08f1bf155 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:19.529489 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c093ff2-aeaa-4a89-b07e-88e08f1bf155 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:19.529754 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c093ff2-aeaa-4a89-b07e-88e08f1bf155 None admin] PATH_INFO: `/v3/endpoints` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:19.530441 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c093ff2-aeaa-4a89-b07e-88e08f1bf155 None admin] RBAC: Adding query filter params (interface=public, service_id=72811bbe485b4623b950c5f1b43a4b6f, region_id=RegionOne) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:19.530741 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c093ff2-aeaa-4a89-b07e-88e08f1bf155 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:19.532125 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c093ff2-aeaa-4a89-b07e-88e08f1bf155 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:19.542844 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 443/901] 10.222.0.27 () {56 vars in 1251 bytes} [Sun Sep 11 00:40:19 2022] GET /identity/v3/endpoints?service_id=72811bbe485b4623b950c5f1b43a4b6f&interface=public®ion_id=RegionOne => generated 194 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:19.542844 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c093ff2-aeaa-4a89-b07e-88e08f1bf155 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:19.585728 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a47febaf-a675-4337-8919-0bd4114666e9 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:19.623980 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73452dee-6cbd-4b03-b218-256eb40c1eae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:19.624592 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73452dee-6cbd-4b03-b218-256eb40c1eae None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:19.624872 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73452dee-6cbd-4b03-b218-256eb40c1eae None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:19.625141 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73452dee-6cbd-4b03-b218-256eb40c1eae None admin] PATH_INFO: `/v3/services` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:19.625841 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73452dee-6cbd-4b03-b218-256eb40c1eae None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:19.626151 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73452dee-6cbd-4b03-b218-256eb40c1eae None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:19.627531 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73452dee-6cbd-4b03-b218-256eb40c1eae None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:19.634112 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 459/902] 10.222.0.27 () {56 vars in 1114 bytes} [Sun Sep 11 00:40:19 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:19.636363 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73452dee-6cbd-4b03-b218-256eb40c1eae None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:19.654891 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c093ff2-aeaa-4a89-b07e-88e08f1bf155 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:19.687929 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:19.694337 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79ee879c-d560-41d6-8962-d1fe225a170f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:19.695021 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ee879c-d560-41d6-8962-d1fe225a170f None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:19.695295 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ee879c-d560-41d6-8962-d1fe225a170f None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:19.695558 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ee879c-d560-41d6-8962-d1fe225a170f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:19.714850 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:19.748541 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73452dee-6cbd-4b03-b218-256eb40c1eae None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:19.766761 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79ee879c-d560-41d6-8962-d1fe225a170f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:19.768532 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79ee879c-d560-41d6-8962-d1fe225a170f None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:19.786658 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5a64ab-3ef3-4ad5-bb26-b7d11c97b35d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:19.788059 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5a64ab-3ef3-4ad5-bb26-b7d11c97b35d None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:19.788412 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5a64ab-3ef3-4ad5-bb26-b7d11c97b35d None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:19.788681 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5a64ab-3ef3-4ad5-bb26-b7d11c97b35d None admin] PATH_INFO: `/v3/services` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:19.789292 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5a64ab-3ef3-4ad5-bb26-b7d11c97b35d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:19.790648 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5a64ab-3ef3-4ad5-bb26-b7d11c97b35d None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:19.796663 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 460/903] 10.222.0.27 () {56 vars in 1087 bytes} [Sun Sep 11 00:40:19 2022] GET /identity/v3/services => generated 1497 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:40:19.797437 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5a64ab-3ef3-4ad5-bb26-b7d11c97b35d None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:19.898365 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 444/904] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:40:19 2022] GET /identity/v3/auth/tokens => generated 1474 bytes in 359 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 11 00:40:19.903891 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79ee879c-d560-41d6-8962-d1fe225a170f None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:19.938463 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:19.945814 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5a64ab-3ef3-4ad5-bb26-b7d11c97b35d None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:19.993858 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99c44ed-0ba3-49de-83d3-c034c2345006 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:19.994095 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99c44ed-0ba3-49de-83d3-c034c2345006 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:19.994339 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99c44ed-0ba3-49de-83d3-c034c2345006 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:19.994505 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99c44ed-0ba3-49de-83d3-c034c2345006 None admin] PATH_INFO: `/v3/services` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:19.995141 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99c44ed-0ba3-49de-83d3-c034c2345006 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:19.995436 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99c44ed-0ba3-49de-83d3-c034c2345006 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:19.997091 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99c44ed-0ba3-49de-83d3-c034c2345006 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:19.999959 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:20.006058 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 461/905] 10.222.0.27 () {56 vars in 1114 bytes} [Sun Sep 11 00:40:19 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:20.013535 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99c44ed-0ba3-49de-83d3-c034c2345006 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:20.037588 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79ee879c-d560-41d6-8962-d1fe225a170f None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:20.091286 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98d15620-1727-4880-85a6-254c0084eb61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:20.092836 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d15620-1727-4880-85a6-254c0084eb61 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:20.093590 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d15620-1727-4880-85a6-254c0084eb61 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:20.094085 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d15620-1727-4880-85a6-254c0084eb61 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:20.112585 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98d15620-1727-4880-85a6-254c0084eb61 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:20.114908 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98d15620-1727-4880-85a6-254c0084eb61 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:20.128564 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99c44ed-0ba3-49de-83d3-c034c2345006 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:20.137354 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-98d15620-1727-4880-85a6-254c0084eb61 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:40:20.137354 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:20.137354 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:20.137354 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:20.137354 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:20.137354 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:20.137354 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:20.137354 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:20.137354 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:20.137354 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:20.137354 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:20.137354 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:20.137354 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:40:20.137354 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:40:20.137354 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:40:20.137354 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:40:20.138356 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:20.138356 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:20.138356 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:20.138356 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:20.138356 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:20.138356 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:20.138356 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:20.138356 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:20.138356 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:20.138356 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:20.138356 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:20.138356 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:20.138356 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:20.138356 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:20.138356 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:20.138356 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:20.138356 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:40:20.139288 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:40:20.139288 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:40:20.139288 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:20.142368 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 445/906] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:40:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:20.147724 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98d15620-1727-4880-85a6-254c0084eb61 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:20.169219 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57300632-e809-4f62-9c54-9cacd5a4f62a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:20.169834 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57300632-e809-4f62-9c54-9cacd5a4f62a None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:20.170121 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57300632-e809-4f62-9c54-9cacd5a4f62a None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:20.170388 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57300632-e809-4f62-9c54-9cacd5a4f62a None admin] PATH_INFO: `/v3/endpoints` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:20.171538 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57300632-e809-4f62-9c54-9cacd5a4f62a None admin] RBAC: Adding query filter params (interface=public, service_id=1f597397734c4704818d9f153494fe4e, region_id=RegionOne) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:20.171852 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57300632-e809-4f62-9c54-9cacd5a4f62a None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:20.173511 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57300632-e809-4f62-9c54-9cacd5a4f62a None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:20.184433 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 462/907] 10.222.0.27 () {56 vars in 1251 bytes} [Sun Sep 11 00:40:20 2022] GET /identity/v3/endpoints?service_id=1f597397734c4704818d9f153494fe4e&interface=public®ion_id=RegionOne => generated 194 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:20.185316 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57300632-e809-4f62-9c54-9cacd5a4f62a None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:20.267328 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:20.272999 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:20.297111 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98d15620-1727-4880-85a6-254c0084eb61 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:20.312663 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57300632-e809-4f62-9c54-9cacd5a4f62a None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:20.345424 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85cf6863-bed2-4b70-bae3-f0cbffdd9510 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:20.346370 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85cf6863-bed2-4b70-bae3-f0cbffdd9510 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:20.346919 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85cf6863-bed2-4b70-bae3-f0cbffdd9510 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:20.347401 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85cf6863-bed2-4b70-bae3-f0cbffdd9510 None admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:20.348236 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85cf6863-bed2-4b70-bae3-f0cbffdd9510 None admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:20.348746 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85cf6863-bed2-4b70-bae3-f0cbffdd9510 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:20.351379 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85cf6863-bed2-4b70-bae3-f0cbffdd9510 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:20.354641 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-262e0eea-f25e-4029-8848-c07011f511ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:20.355336 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-262e0eea-f25e-4029-8848-c07011f511ad None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:20.355620 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-262e0eea-f25e-4029-8848-c07011f511ad None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:20.355893 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-262e0eea-f25e-4029-8848-c07011f511ad None admin] PATH_INFO: `/v3/services` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:20.356507 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-262e0eea-f25e-4029-8848-c07011f511ad None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:20.357854 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-262e0eea-f25e-4029-8848-c07011f511ad None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:20.366243 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 446/908] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:40:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:20.367057 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 463/909] 10.222.0.27 () {56 vars in 1087 bytes} [Sun Sep 11 00:40:20 2022] GET /identity/v3/services => generated 1497 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:40:20.371721 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85cf6863-bed2-4b70-bae3-f0cbffdd9510 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:20.492678 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85cf6863-bed2-4b70-bae3-f0cbffdd9510 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:20.533730 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9e03841-02ef-467f-b9ae-cd6bf00dc31b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:20.534733 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e03841-02ef-467f-b9ae-cd6bf00dc31b None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:20.535561 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e03841-02ef-467f-b9ae-cd6bf00dc31b None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:20.536040 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e03841-02ef-467f-b9ae-cd6bf00dc31b None admin] PATH_INFO: `/v3/projects/service` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:20.545068 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9e03841-02ef-467f-b9ae-cd6bf00dc31b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:20.546827 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9e03841-02ef-467f-b9ae-cd6bf00dc31b None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:20.559499 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-d9e03841-02ef-467f-b9ae-cd6bf00dc31b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:40:20.559499 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:20.559499 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:20.559499 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:20.559499 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:20.559499 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:20.559499 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:20.559499 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:20.559499 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:20.559499 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:20.559499 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:20.559499 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:20.559499 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 11 00:40:20.559499 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 11 00:40:20.559499 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 11 00:40:20.560492 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 11 00:40:20.560492 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:20.560492 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:20.560492 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:20.560492 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:20.560492 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:20.560492 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:20.560492 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:20.560492 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:20.560492 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:20.560492 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:20.560492 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:20.560492 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:20.560492 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:20.560492 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:20.560492 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:20.560492 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:20.561377 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 11 00:40:20.561377 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 11 00:40:20.561377 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 11 00:40:20.561377 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 11 00:40:20.561377 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 11 00:40:20.561377 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 11 00:40:20.561377 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:40:20.561377 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:20.563738 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 447/910] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:40:20 2022] GET /identity/v3/projects/service => generated 88 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:40:20.567016 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-262e0eea-f25e-4029-8848-c07011f511ad None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:20.672892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-262e0eea-f25e-4029-8848-c07011f511ad None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:20.701453 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:20.710911 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c138b6d-c24f-4cbc-a287-6a10512f873d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:20.711950 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c138b6d-c24f-4cbc-a287-6a10512f873d None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:20.712474 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c138b6d-c24f-4cbc-a287-6a10512f873d None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:20.713126 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c138b6d-c24f-4cbc-a287-6a10512f873d None admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:20.713814 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c138b6d-c24f-4cbc-a287-6a10512f873d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:20.714116 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c138b6d-c24f-4cbc-a287-6a10512f873d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:20.715578 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c138b6d-c24f-4cbc-a287-6a10512f873d None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:20.728904 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 464/911] 10.222.0.27 () {56 vars in 1148 bytes} [Sun Sep 11 00:40:20 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:20.734419 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9e03841-02ef-467f-b9ae-cd6bf00dc31b None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:20.739317 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:20.849395 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9e03841-02ef-467f-b9ae-cd6bf00dc31b None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:20.886505 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe0de1e2-bf59-4ee1-a25f-5b17bf2400c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:20.890838 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0de1e2-bf59-4ee1-a25f-5b17bf2400c7 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:20.890838 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0de1e2-bf59-4ee1-a25f-5b17bf2400c7 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:20.890838 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0de1e2-bf59-4ee1-a25f-5b17bf2400c7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:20.890838 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe0de1e2-bf59-4ee1-a25f-5b17bf2400c7 None admin] RBAC: Adding query filter params (role.id=a6b5856c8f5c4144b39cab1b2916ac93, scope.project.id=b504a5058bc6474c8cc077dfeec16799, user.id=9fc4274836844ae4ad5d16d74c36f671) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:20.890838 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe0de1e2-bf59-4ee1-a25f-5b17bf2400c7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:20.890838 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe0de1e2-bf59-4ee1-a25f-5b17bf2400c7 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:20.896867 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 448/912] 10.222.0.27 () {56 vars in 1374 bytes} [Sun Sep 11 00:40:20 2022] GET /identity/v3/role_assignments?user.id=9fc4274836844ae4ad5d16d74c36f671&scope.project.id=b504a5058bc6474c8cc077dfeec16799&role.id=a6b5856c8f5c4144b39cab1b2916ac93 => generated 259 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:21.329532 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f474d10a-3f4c-42a7-908c-2488779b8bc8 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:21.330054 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f474d10a-3f4c-42a7-908c-2488779b8bc8 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:21.330054 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f474d10a-3f4c-42a7-908c-2488779b8bc8 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:21.331655 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 465/913] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:40:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:40:21.337242 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f5f421-a5fc-4a5d-a82c-e4dca5b88efd None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:21.337651 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f5f421-a5fc-4a5d-a82c-e4dca5b88efd None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:21.337834 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f5f421-a5fc-4a5d-a82c-e4dca5b88efd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:21.356787 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-d8f5f421-a5fc-4a5d-a82c-e4dca5b88efd None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:40:21.357892 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-d8f5f421-a5fc-4a5d-a82c-e4dca5b88efd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:40:21.422604 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:21.428786 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 449/914] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:40:21 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:40:21.697937 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f474d10a-3f4c-42a7-908c-2488779b8bc8 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:21.807303 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f474d10a-3f4c-42a7-908c-2488779b8bc8 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:21.841965 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fef31cc-686b-42f6-b116-9f8ea0648c56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:21.842599 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fef31cc-686b-42f6-b116-9f8ea0648c56 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:21.842897 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fef31cc-686b-42f6-b116-9f8ea0648c56 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:21.843173 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fef31cc-686b-42f6-b116-9f8ea0648c56 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:21.843759 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fef31cc-686b-42f6-b116-9f8ea0648c56 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:21.845120 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fef31cc-686b-42f6-b116-9f8ea0648c56 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:21.854630 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-3fef31cc-686b-42f6-b116-9f8ea0648c56 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 11 00:40:21.854630 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:21.854630 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 11 00:40:21.854630 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 11 00:40:21.854630 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 11 00:40:21.854630 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 11 00:40:21.854630 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 11 00:40:21.854630 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 11 00:40:21.854630 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 11 00:40:21.854630 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:21.854630 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 11 00:40:21.854630 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:21.854630 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:21.854630 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:21.854630 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:21.854630 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:21.854630 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:21.855689 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:21.855689 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:21.855689 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:21.855689 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:21.855689 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:21.855689 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:21.855689 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 11 00:40:21.855689 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 11 00:40:21.855689 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 11 00:40:21.855689 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 11 00:40:21.855689 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:21.855689 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:21.855689 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:21.855689 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:21.855689 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:21.855689 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:21.856565 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:21.856565 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:21.856565 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:21.856565 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:21.856565 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:21.856565 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:21.856565 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:21.856565 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:21.856565 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:21.856565 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:21.856565 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 11 00:40:21.856565 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 11 00:40:21.856565 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 11 00:40:21.856565 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:21.856565 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 466/915] 10.222.0.27 () {56 vars in 1126 bytes} [Sun Sep 11 00:40:21 2022] GET /identity/v3/services/object-store => generated 93 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:21.862231 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8f5f421-a5fc-4a5d-a82c-e4dca5b88efd None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:21.961862 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8f5f421-a5fc-4a5d-a82c-e4dca5b88efd None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:22.000551 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48793222-644b-4c94-9348-c1fd142dfd23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:22.001169 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48793222-644b-4c94-9348-c1fd142dfd23 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:22.001443 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48793222-644b-4c94-9348-c1fd142dfd23 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:22.001712 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48793222-644b-4c94-9348-c1fd142dfd23 None admin] PATH_INFO: `/v3/services` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:22.002349 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48793222-644b-4c94-9348-c1fd142dfd23 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:22.002646 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48793222-644b-4c94-9348-c1fd142dfd23 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:22.004070 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48793222-644b-4c94-9348-c1fd142dfd23 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:22.010977 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 450/916] 10.222.0.27 () {56 vars in 1122 bytes} [Sun Sep 11 00:40:21 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:22.016398 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fef31cc-686b-42f6-b116-9f8ea0648c56 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:22.074384 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:22.116789 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030a2072-a4f6-4ee6-a806-59ad7203feb9 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:22.117103 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030a2072-a4f6-4ee6-a806-59ad7203feb9 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:22.117332 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030a2072-a4f6-4ee6-a806-59ad7203feb9 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:22.118813 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 451/917] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:40:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:40:22.124087 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e625d31-8d06-4b6f-ab3b-2b0c87b83f94 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:22.124614 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e625d31-8d06-4b6f-ab3b-2b0c87b83f94 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:22.125087 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e625d31-8d06-4b6f-ab3b-2b0c87b83f94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:22.136475 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fef31cc-686b-42f6-b116-9f8ea0648c56 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:22.147947 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-3e625d31-8d06-4b6f-ab3b-2b0c87b83f94 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:40:22.150170 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-3e625d31-8d06-4b6f-ab3b-2b0c87b83f94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:40:22.178226 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e982c27e-3a68-4f46-a9cf-763f982d2dce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:22.178923 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e982c27e-3a68-4f46-a9cf-763f982d2dce None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:22.179702 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e982c27e-3a68-4f46-a9cf-763f982d2dce None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:22.179985 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e982c27e-3a68-4f46-a9cf-763f982d2dce None admin] PATH_INFO: `/v3/services` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:22.180619 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e982c27e-3a68-4f46-a9cf-763f982d2dce None admin] RBAC: Adding query filter params (type=object-store) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:22.181127 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e982c27e-3a68-4f46-a9cf-763f982d2dce None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:22.182756 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e982c27e-3a68-4f46-a9cf-763f982d2dce None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:22.189868 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 467/918] 10.222.0.27 () {56 vars in 1122 bytes} [Sun Sep 11 00:40:22 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:22.196413 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e982c27e-3a68-4f46-a9cf-763f982d2dce None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:22.226702 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:22.238015 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 452/919] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:40:22 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 11 00:40:22.312697 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e982c27e-3a68-4f46-a9cf-763f982d2dce None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:22.329715 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:22.348357 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce39a284-5f48-414e-93ba-2185ac4eff0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:22.348969 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce39a284-5f48-414e-93ba-2185ac4eff0e None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:22.349269 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce39a284-5f48-414e-93ba-2185ac4eff0e None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:22.349624 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce39a284-5f48-414e-93ba-2185ac4eff0e None admin] PATH_INFO: `/v3/endpoints` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:22.350469 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce39a284-5f48-414e-93ba-2185ac4eff0e None admin] RBAC: Adding query filter params (interface=public, service_id=b4c75091a7f04c45ae258fb24687ccc0, region_id=RegionOne) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:22.350814 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce39a284-5f48-414e-93ba-2185ac4eff0e None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:22.352183 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce39a284-5f48-414e-93ba-2185ac4eff0e None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:22.357753 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 468/920] 10.222.0.27 () {56 vars in 1251 bytes} [Sun Sep 11 00:40:22 2022] GET /identity/v3/endpoints?service_id=b4c75091a7f04c45ae258fb24687ccc0&interface=public®ion_id=RegionOne => generated 194 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:22.363486 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e625d31-8d06-4b6f-ab3b-2b0c87b83f94 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:22.470721 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e625d31-8d06-4b6f-ab3b-2b0c87b83f94 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:22.508362 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deede8d8-95f0-4398-a7ff-35852d5a877b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:22.509356 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deede8d8-95f0-4398-a7ff-35852d5a877b None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:22.509954 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deede8d8-95f0-4398-a7ff-35852d5a877b None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:22.510434 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deede8d8-95f0-4398-a7ff-35852d5a877b None admin] PATH_INFO: `/v3/services` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:22.511068 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce39a284-5f48-414e-93ba-2185ac4eff0e None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:22.511660 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deede8d8-95f0-4398-a7ff-35852d5a877b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:22.513384 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deede8d8-95f0-4398-a7ff-35852d5a877b None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:22.521770 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 453/921] 10.222.0.27 () {56 vars in 1087 bytes} [Sun Sep 11 00:40:22 2022] GET /identity/v3/services => generated 1497 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:40:22.626930 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce39a284-5f48-414e-93ba-2185ac4eff0e None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:22.632086 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b691a80c-476f-4d9b-8e8f-e90ca7534653 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:22.632370 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b691a80c-476f-4d9b-8e8f-e90ca7534653 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:22.632637 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b691a80c-476f-4d9b-8e8f-e90ca7534653 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:22.634264 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 454/922] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:40:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 11 00:40:22.639858 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2d505b-83f1-497b-995c-02fdbf31b394 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:22.640152 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2d505b-83f1-497b-995c-02fdbf31b394 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:22.640406 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2d505b-83f1-497b-995c-02fdbf31b394 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:22.660665 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-2e2d505b-83f1-497b-995c-02fdbf31b394 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:40:22.663270 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-2e2d505b-83f1-497b-995c-02fdbf31b394 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:40:22.671567 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68425e3-059f-48ab-a26c-d48331ac50e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:22.672610 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68425e3-059f-48ab-a26c-d48331ac50e0 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:22.673216 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68425e3-059f-48ab-a26c-d48331ac50e0 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:22.673496 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68425e3-059f-48ab-a26c-d48331ac50e0 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:22.674088 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b68425e3-059f-48ab-a26c-d48331ac50e0 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:22.675513 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b68425e3-059f-48ab-a26c-d48331ac50e0 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:22.686276 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-b68425e3-059f-48ab-a26c-d48331ac50e0 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 11 00:40:22.686276 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:22.686276 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 11 00:40:22.686276 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 11 00:40:22.686276 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 11 00:40:22.686276 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 11 00:40:22.686276 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 11 00:40:22.686276 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 11 00:40:22.686276 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 11 00:40:22.686276 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:22.686276 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 11 00:40:22.686276 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:22.686276 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:22.686276 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:22.686276 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:22.686276 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:22.687374 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:22.687374 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:22.687374 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:22.687374 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:22.687374 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:22.687374 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:22.687374 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:22.687374 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 11 00:40:22.687374 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 11 00:40:22.687374 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 11 00:40:22.687374 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 11 00:40:22.687374 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:22.687374 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:22.687374 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:22.687374 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:22.687374 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:22.687374 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:22.688278 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:22.688278 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:22.688278 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:22.688278 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:22.688278 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:22.688278 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:22.688278 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:22.688278 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:22.688278 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:22.688278 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:22.688278 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 11 00:40:22.688278 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 11 00:40:22.688278 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 11 00:40:22.688278 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:22.688278 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 469/923] 10.222.0.27 () {56 vars in 1132 bytes} [Sun Sep 11 00:40:22 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:22.694510 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68425e3-059f-48ab-a26c-d48331ac50e0 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:22.738499 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:22.751020 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:22.760206 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 455/924] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:40:22 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:40:22.815182 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68425e3-059f-48ab-a26c-d48331ac50e0 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:22.852082 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-100e725e-dce4-41f9-8a56-ebd8471c7ef1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:22.852993 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100e725e-dce4-41f9-8a56-ebd8471c7ef1 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:22.853478 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100e725e-dce4-41f9-8a56-ebd8471c7ef1 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:22.853899 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100e725e-dce4-41f9-8a56-ebd8471c7ef1 None admin] PATH_INFO: `/v3/services` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:22.854695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-100e725e-dce4-41f9-8a56-ebd8471c7ef1 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:22.855171 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-100e725e-dce4-41f9-8a56-ebd8471c7ef1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:22.856685 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-100e725e-dce4-41f9-8a56-ebd8471c7ef1 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:22.865930 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 470/925] 10.222.0.27 () {56 vars in 1126 bytes} [Sun Sep 11 00:40:22 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:40:22.871127 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e2d505b-83f1-497b-995c-02fdbf31b394 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:22.979814 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e2d505b-83f1-497b-995c-02fdbf31b394 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:23.015522 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-266e37eb-1940-4aab-8504-78ee8d6727a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:23.016052 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-266e37eb-1940-4aab-8504-78ee8d6727a6 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:23.016330 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-266e37eb-1940-4aab-8504-78ee8d6727a6 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:23.016602 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-266e37eb-1940-4aab-8504-78ee8d6727a6 None admin] PATH_INFO: `/v3/services` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:23.017242 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-266e37eb-1940-4aab-8504-78ee8d6727a6 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:23.017545 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-266e37eb-1940-4aab-8504-78ee8d6727a6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:23.018960 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-266e37eb-1940-4aab-8504-78ee8d6727a6 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:23.025638 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 456/926] 10.222.0.27 () {56 vars in 1126 bytes} [Sun Sep 11 00:40:22 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:23.026473 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-100e725e-dce4-41f9-8a56-ebd8471c7ef1 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:23.030648 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-266e37eb-1940-4aab-8504-78ee8d6727a6 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:23.092521 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:23.163086 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-266e37eb-1940-4aab-8504-78ee8d6727a6 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:23.165091 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-100e725e-dce4-41f9-8a56-ebd8471c7ef1 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:23.174918 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:23.202827 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a7b1687-54a9-426d-83ad-79bf5daedfaa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:23.203824 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7b1687-54a9-426d-83ad-79bf5daedfaa None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:23.204316 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7b1687-54a9-426d-83ad-79bf5daedfaa None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:23.204796 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7b1687-54a9-426d-83ad-79bf5daedfaa None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:23.205592 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a7b1687-54a9-426d-83ad-79bf5daedfaa None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:23.207234 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a7b1687-54a9-426d-83ad-79bf5daedfaa None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:23.211127 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8392dfe-d8ab-4e30-b960-b8b988335513 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:23.212139 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8392dfe-d8ab-4e30-b960-b8b988335513 None admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:23.212422 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8392dfe-d8ab-4e30-b960-b8b988335513 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:23.212695 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8392dfe-d8ab-4e30-b960-b8b988335513 None admin] PATH_INFO: `/v3/endpoints` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:23.213755 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8392dfe-d8ab-4e30-b960-b8b988335513 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:23.215166 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8392dfe-d8ab-4e30-b960-b8b988335513 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:23.219885 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-2a7b1687-54a9-426d-83ad-79bf5daedfaa None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 11 00:40:23.219885 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:23.219885 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 11 00:40:23.219885 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 11 00:40:23.219885 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 11 00:40:23.219885 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 11 00:40:23.219885 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 11 00:40:23.219885 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 11 00:40:23.219885 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 11 00:40:23.219885 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:23.219885 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 11 00:40:23.219885 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:23.219885 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:23.219885 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:23.219885 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:23.219885 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:23.219885 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:23.220894 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:23.220894 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:23.220894 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:23.220894 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:23.220894 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:23.220894 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:23.220894 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 11 00:40:23.220894 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 11 00:40:23.220894 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 11 00:40:23.220894 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 11 00:40:23.220894 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:23.220894 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:23.220894 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:23.220894 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:23.220894 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:23.220894 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:23.221769 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:23.221769 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:23.221769 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:23.221769 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:23.221769 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:23.221769 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:23.221769 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:23.221769 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:23.221769 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:23.221769 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:23.221769 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 11 00:40:23.221769 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 11 00:40:23.221769 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 11 00:40:23.221769 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:23.224628 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 471/927] 10.222.0.27 () {56 vars in 1114 bytes} [Sun Sep 11 00:40:23 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:23.230367 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a7b1687-54a9-426d-83ad-79bf5daedfaa None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:23.240817 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 457/928] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:40:23 2022] POST /identity/v3/endpoints => generated 347 bytes in 210 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:40:23.253476 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eac7a9f-65cc-4f90-8e75-dbf5b20a3b7c None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:23.253786 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eac7a9f-65cc-4f90-8e75-dbf5b20a3b7c None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:23.254033 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eac7a9f-65cc-4f90-8e75-dbf5b20a3b7c None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:23.260093 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 458/929] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:40:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 11 00:40:23.262013 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832b6627-ab77-474a-ad28-56314c62bdb2 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:23.262305 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832b6627-ab77-474a-ad28-56314c62bdb2 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:23.262580 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832b6627-ab77-474a-ad28-56314c62bdb2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:23.283948 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-832b6627-ab77-474a-ad28-56314c62bdb2 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:40:23.285333 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-832b6627-ab77-474a-ad28-56314c62bdb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:40:23.350690 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a7b1687-54a9-426d-83ad-79bf5daedfaa None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:23.377558 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 459/930] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:40:23 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:40:23.389908 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29b6cc22-b394-464d-88d0-3de460ce460f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:23.390528 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b6cc22-b394-464d-88d0-3de460ce460f None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:23.390827 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b6cc22-b394-464d-88d0-3de460ce460f None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:23.391106 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b6cc22-b394-464d-88d0-3de460ce460f None admin] PATH_INFO: `/v3/services` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:23.391750 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29b6cc22-b394-464d-88d0-3de460ce460f None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:23.392052 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29b6cc22-b394-464d-88d0-3de460ce460f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:23.393447 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29b6cc22-b394-464d-88d0-3de460ce460f None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:23.399617 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 472/931] 10.222.0.27 () {56 vars in 1114 bytes} [Sun Sep 11 00:40:23 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:23.408250 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832b6627-ab77-474a-ad28-56314c62bdb2 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:23.443785 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:23.524994 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832b6627-ab77-474a-ad28-56314c62bdb2 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:23.561858 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d614d3f5-f94a-4b7b-9164-bdcfa975b01e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:23.562385 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d614d3f5-f94a-4b7b-9164-bdcfa975b01e None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:23.562664 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d614d3f5-f94a-4b7b-9164-bdcfa975b01e None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:23.562966 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d614d3f5-f94a-4b7b-9164-bdcfa975b01e None admin] PATH_INFO: `/v3/services` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:23.563604 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d614d3f5-f94a-4b7b-9164-bdcfa975b01e None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:23.563906 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d614d3f5-f94a-4b7b-9164-bdcfa975b01e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:23.565276 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d614d3f5-f94a-4b7b-9164-bdcfa975b01e None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:23.571475 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 460/932] 10.222.0.27 () {56 vars in 1114 bytes} [Sun Sep 11 00:40:23 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:23.577382 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29b6cc22-b394-464d-88d0-3de460ce460f None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:23.612921 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:23.685455 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29b6cc22-b394-464d-88d0-3de460ce460f None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:23.697181 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d614d3f5-f94a-4b7b-9164-bdcfa975b01e None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:23.726844 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-669b35ab-5508-44d8-b4ce-72a36d606095 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:23.727580 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669b35ab-5508-44d8-b4ce-72a36d606095 None admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:23.727858 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669b35ab-5508-44d8-b4ce-72a36d606095 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:23.728127 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669b35ab-5508-44d8-b4ce-72a36d606095 None admin] PATH_INFO: `/v3/endpoints` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:23.728833 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-669b35ab-5508-44d8-b4ce-72a36d606095 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:23.730179 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-669b35ab-5508-44d8-b4ce-72a36d606095 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:23.758861 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 473/933] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:40:23 2022] POST /identity/v3/endpoints => generated 346 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:40:23.825493 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:23.832131 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d614d3f5-f94a-4b7b-9164-bdcfa975b01e None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:23.882781 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac6fc6fd-0e12-4da3-9acb-2a5bc2c2d143 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:23.883441 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac6fc6fd-0e12-4da3-9acb-2a5bc2c2d143 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:23.883715 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac6fc6fd-0e12-4da3-9acb-2a5bc2c2d143 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:23.883983 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac6fc6fd-0e12-4da3-9acb-2a5bc2c2d143 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:23.895997 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac6fc6fd-0e12-4da3-9acb-2a5bc2c2d143 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:40:23.901894 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-ac6fc6fd-0e12-4da3-9acb-2a5bc2c2d143 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 11 00:40:23.901894 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:23.901894 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:23.901894 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:23.901894 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:23.901894 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:23.901894 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:23.901894 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:23.901894 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:23.901894 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:23.901894 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:23.901894 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:23.901894 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:40:23.901894 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:40:23.901894 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:40:23.901894 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:40:23.902927 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:40:23.902927 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:40:23.902927 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:40:23.902927 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:40:23.902927 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:23.902927 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:23.902927 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:23.902927 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:23.902927 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:23.902927 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:23.902927 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:23.902927 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:23.902927 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:23.902927 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:23.902927 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:23.902927 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:23.903783 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:23.903783 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:23.903783 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:23.903783 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:23.903783 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:40:23.903783 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:40:23.903783 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:40:23.903783 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:40:23.903783 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:40:23.903783 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:40:23.903783 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 11 00:40:23.903783 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:23.903783 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 461/934] 10.222.0.27 () {56 vars in 1120 bytes} [Sun Sep 11 00:40:23 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:40:23.910835 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-669b35ab-5508-44d8-b4ce-72a36d606095 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:24.015408 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:24.028069 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-669b35ab-5508-44d8-b4ce-72a36d606095 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:24.062736 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56171a91-9787-4210-bb83-524a2d9f456b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:24.063372 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56171a91-9787-4210-bb83-524a2d9f456b None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:24.063646 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56171a91-9787-4210-bb83-524a2d9f456b None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:24.063916 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56171a91-9787-4210-bb83-524a2d9f456b None admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:24.064583 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56171a91-9787-4210-bb83-524a2d9f456b None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:24.064884 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56171a91-9787-4210-bb83-524a2d9f456b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:24.066249 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56171a91-9787-4210-bb83-524a2d9f456b None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:24.075256 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 474/935] 10.222.0.27 () {56 vars in 1115 bytes} [Sun Sep 11 00:40:23 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:24.081581 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac6fc6fd-0e12-4da3-9acb-2a5bc2c2d143 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:24.195368 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac6fc6fd-0e12-4da3-9acb-2a5bc2c2d143 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:24.237562 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3bf4d5d-9c3c-459d-a6ac-64520f65b120 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:24.238594 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bf4d5d-9c3c-459d-a6ac-64520f65b120 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:24.239095 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bf4d5d-9c3c-459d-a6ac-64520f65b120 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:24.239588 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bf4d5d-9c3c-459d-a6ac-64520f65b120 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:24.253067 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3bf4d5d-9c3c-459d-a6ac-64520f65b120 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:24.255821 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3bf4d5d-9c3c-459d-a6ac-64520f65b120 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:24.276518 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-e3bf4d5d-9c3c-459d-a6ac-64520f65b120 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:40:24.276518 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:24.276518 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:24.276518 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:24.276518 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:24.276518 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:24.276518 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:24.276518 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:24.276518 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:24.276518 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:24.276518 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:24.276518 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:24.276518 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:40:24.276518 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:40:24.276518 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:40:24.276518 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:40:24.277555 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:24.277555 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:24.277555 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:24.277555 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:24.277555 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:24.277555 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:24.277555 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:24.277555 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:24.277555 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:24.277555 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:24.277555 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:24.277555 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:24.277555 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:24.277555 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:24.277555 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:24.277555 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:24.277555 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:40:24.278466 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:40:24.278466 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:40:24.278466 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:24.280580 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 462/936] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:40:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:40:24.287436 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56171a91-9787-4210-bb83-524a2d9f456b None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:24.399465 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56171a91-9787-4210-bb83-524a2d9f456b None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:24.442362 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a753620e-c5d7-4866-a166-38a1006b20ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:24.443108 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a753620e-c5d7-4866-a166-38a1006b20ed None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:24.443457 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a753620e-c5d7-4866-a166-38a1006b20ed None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:24.443730 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a753620e-c5d7-4866-a166-38a1006b20ed None admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:24.444379 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a753620e-c5d7-4866-a166-38a1006b20ed None admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:24.444742 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a753620e-c5d7-4866-a166-38a1006b20ed None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:24.446207 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a753620e-c5d7-4866-a166-38a1006b20ed None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:24.460939 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 475/937] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:40:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:24.466758 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3bf4d5d-9c3c-459d-a6ac-64520f65b120 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:24.471631 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:24.581816 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3bf4d5d-9c3c-459d-a6ac-64520f65b120 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:24.620220 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f900769f-76ad-46e1-85dd-32a9ca03fc32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:24.620850 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f900769f-76ad-46e1-85dd-32a9ca03fc32 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:24.621302 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f900769f-76ad-46e1-85dd-32a9ca03fc32 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:24.621578 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f900769f-76ad-46e1-85dd-32a9ca03fc32 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:24.631009 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f900769f-76ad-46e1-85dd-32a9ca03fc32 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:24.632899 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f900769f-76ad-46e1-85dd-32a9ca03fc32 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:24.645020 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-f900769f-76ad-46e1-85dd-32a9ca03fc32 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 11 00:40:24.645020 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:24.645020 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:24.645020 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:24.645020 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:24.645020 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:24.645020 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:24.645020 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:24.645020 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:24.645020 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:24.645020 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:24.645020 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:24.645020 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 11 00:40:24.645020 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 11 00:40:24.645020 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 11 00:40:24.645020 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 11 00:40:24.646552 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:24.646552 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:24.646552 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 11 00:40:24.646552 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:40:24.646552 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 11 00:40:24.646552 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:40:24.646552 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:24.646552 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:24.646552 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:24.646552 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:24.646552 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:24.646552 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:24.646552 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:24.646552 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:24.646552 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:24.646552 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:24.646552 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:24.646552 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:24.648214 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:24.648214 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:24.648214 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 11 00:40:24.648214 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 11 00:40:24.648214 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 11 00:40:24.648214 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 11 00:40:24.648214 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 11 00:40:24.648214 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 11 00:40:24.648214 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 11 00:40:24.648214 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:24.648214 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 463/938] 10.222.0.27 () {56 vars in 1117 bytes} [Sun Sep 11 00:40:24 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:40:24.655101 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a753620e-c5d7-4866-a166-38a1006b20ed None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:24.674207 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:24.775706 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a753620e-c5d7-4866-a166-38a1006b20ed None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:24.816837 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab1afe4-8f55-4ce0-bbf4-a074003f2724 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:24.817947 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab1afe4-8f55-4ce0-bbf4-a074003f2724 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:24.818584 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab1afe4-8f55-4ce0-bbf4-a074003f2724 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:24.819145 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab1afe4-8f55-4ce0-bbf4-a074003f2724 None admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:24.820087 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ab1afe4-8f55-4ce0-bbf4-a074003f2724 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:24.820594 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ab1afe4-8f55-4ce0-bbf4-a074003f2724 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:24.822296 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ab1afe4-8f55-4ce0-bbf4-a074003f2724 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:24.842127 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 476/939] 10.222.0.27 () {56 vars in 1149 bytes} [Sun Sep 11 00:40:24 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:40:24.848106 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f900769f-76ad-46e1-85dd-32a9ca03fc32 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:24.915039 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d49263-428a-409f-8b8b-f6086726e490 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:24.915787 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d49263-428a-409f-8b8b-f6086726e490 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:24.916349 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d49263-428a-409f-8b8b-f6086726e490 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:24.918287 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 477/940] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:40:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 11 00:40:24.923705 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:24.924901 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133a9161-a867-4adc-bf48-c009559e832c None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:24.925316 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133a9161-a867-4adc-bf48-c009559e832c None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:24.925675 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133a9161-a867-4adc-bf48-c009559e832c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:24.952321 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-133a9161-a867-4adc-bf48-c009559e832c None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:40:24.954225 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-133a9161-a867-4adc-bf48-c009559e832c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:40:24.971307 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f900769f-76ad-46e1-85dd-32a9ca03fc32 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:25.017165 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d361c0-32a9-43cb-a3fe-ab566d797317 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:25.017890 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d361c0-32a9-43cb-a3fe-ab566d797317 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:25.018238 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d361c0-32a9-43cb-a3fe-ab566d797317 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:25.018575 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d361c0-32a9-43cb-a3fe-ab566d797317 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:25.031479 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d361c0-32a9-43cb-a3fe-ab566d797317 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:25.033043 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d361c0-32a9-43cb-a3fe-ab566d797317 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:25.051173 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-e4d361c0-32a9-43cb-a3fe-ab566d797317 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:40:25.051173 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:25.051173 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:25.051173 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:25.051173 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:25.051173 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:25.051173 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:25.051173 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:25.051173 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:25.051173 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:25.051173 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:25.051173 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:25.051173 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:40:25.051173 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:40:25.051173 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:40:25.051173 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:40:25.052190 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:25.052190 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:25.052190 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:25.052190 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:25.052190 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:25.052190 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:25.052190 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:25.052190 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:25.052190 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:25.052190 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:25.052190 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:25.052190 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:25.052190 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:25.052190 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:25.052190 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:25.052190 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:25.052190 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:40:25.053552 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:40:25.053552 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:40:25.053552 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:25.057488 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 464/941] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:40:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:25.058566 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d361c0-32a9-43cb-a3fe-ab566d797317 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:25.083076 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 478/942] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:40:24 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 11 00:40:25.189749 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d361c0-32a9-43cb-a3fe-ab566d797317 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:25.207744 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:25.228235 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e89b160f-5f59-4990-8d99-4a0891481dad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:25.229208 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89b160f-5f59-4990-8d99-4a0891481dad None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:25.229665 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89b160f-5f59-4990-8d99-4a0891481dad None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:25.230144 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89b160f-5f59-4990-8d99-4a0891481dad None admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:25.230994 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e89b160f-5f59-4990-8d99-4a0891481dad None admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:25.231490 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e89b160f-5f59-4990-8d99-4a0891481dad None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:25.233058 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e89b160f-5f59-4990-8d99-4a0891481dad None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:25.251848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 465/943] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:40:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:25.259318 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-133a9161-a867-4adc-bf48-c009559e832c None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:25.262236 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:25.378380 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-133a9161-a867-4adc-bf48-c009559e832c None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:25.391679 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e89b160f-5f59-4990-8d99-4a0891481dad None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:25.417109 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea7edec-bebc-4f32-8ef4-b9d0ca6eaa80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:25.417789 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea7edec-bebc-4f32-8ef4-b9d0ca6eaa80 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:25.418020 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea7edec-bebc-4f32-8ef4-b9d0ca6eaa80 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:25.418294 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea7edec-bebc-4f32-8ef4-b9d0ca6eaa80 None admin] PATH_INFO: `/v3/projects/service` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:25.425222 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea7edec-bebc-4f32-8ef4-b9d0ca6eaa80 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:25.426672 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea7edec-bebc-4f32-8ef4-b9d0ca6eaa80 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:25.438600 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-eea7edec-bebc-4f32-8ef4-b9d0ca6eaa80 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:40:25.438600 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:25.438600 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:25.438600 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:25.438600 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:25.438600 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:25.438600 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:25.438600 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:25.438600 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:25.438600 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:25.438600 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:25.438600 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:25.438600 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 11 00:40:25.438600 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 11 00:40:25.438600 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 11 00:40:25.439571 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 11 00:40:25.439571 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:25.439571 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:25.439571 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:25.439571 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:25.439571 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:25.439571 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:25.439571 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:25.439571 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:25.439571 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:25.439571 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:25.439571 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:25.439571 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:25.439571 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:25.439571 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:25.439571 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:25.439571 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:25.440543 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 11 00:40:25.440543 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 11 00:40:25.440543 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 11 00:40:25.440543 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 11 00:40:25.440543 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 11 00:40:25.440543 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 11 00:40:25.440543 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:40:25.440543 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:25.440543 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 479/944] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:40:25 2022] GET /identity/v3/projects/service => generated 88 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:25.446366 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea7edec-bebc-4f32-8ef4-b9d0ca6eaa80 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:25.523233 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e89b160f-5f59-4990-8d99-4a0891481dad None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:25.529271 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:25.575089 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79d5b366-690a-43be-b472-b91612095b00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:25.576144 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d5b366-690a-43be-b472-b91612095b00 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:25.576641 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d5b366-690a-43be-b472-b91612095b00 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:25.577094 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d5b366-690a-43be-b472-b91612095b00 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:25.577254 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea7edec-bebc-4f32-8ef4-b9d0ca6eaa80 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:25.578015 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79d5b366-690a-43be-b472-b91612095b00 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:25.579811 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79d5b366-690a-43be-b472-b91612095b00 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:25.590473 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-79d5b366-690a-43be-b472-b91612095b00 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 11 00:40:25.590473 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:25.590473 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 11 00:40:25.590473 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 11 00:40:25.590473 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 11 00:40:25.590473 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 11 00:40:25.590473 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 11 00:40:25.590473 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 11 00:40:25.590473 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 11 00:40:25.590473 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:25.590473 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 11 00:40:25.590473 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:25.590473 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:25.590473 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:25.590473 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:25.590473 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:25.590473 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:25.591574 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:25.591574 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:25.591574 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:25.591574 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:25.591574 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:25.591574 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:25.591574 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 11 00:40:25.591574 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 11 00:40:25.591574 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 11 00:40:25.591574 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 11 00:40:25.591574 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:25.591574 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:25.591574 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:25.591574 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:25.591574 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:25.591574 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:25.592437 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:25.592437 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:25.592437 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:25.592437 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:25.592437 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:25.592437 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:25.592437 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:25.592437 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:25.592437 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:25.592437 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:25.592437 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 11 00:40:25.592437 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 11 00:40:25.592437 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 11 00:40:25.592437 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:25.595169 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 466/945] 10.222.0.27 () {56 vars in 1126 bytes} [Sun Sep 11 00:40:25 2022] GET /identity/v3/services/object-store => generated 93 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:25.604941 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79d5b366-690a-43be-b472-b91612095b00 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:25.621694 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73211c6a-aab0-4b99-9764-0289a2b32339 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:25.622712 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73211c6a-aab0-4b99-9764-0289a2b32339 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:25.623284 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73211c6a-aab0-4b99-9764-0289a2b32339 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:25.623562 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73211c6a-aab0-4b99-9764-0289a2b32339 None admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:25.624334 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73211c6a-aab0-4b99-9764-0289a2b32339 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:25.624643 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73211c6a-aab0-4b99-9764-0289a2b32339 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:25.626268 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73211c6a-aab0-4b99-9764-0289a2b32339 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:25.644317 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 480/946] 10.222.0.27 () {56 vars in 1148 bytes} [Sun Sep 11 00:40:25 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:40:25.651573 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73211c6a-aab0-4b99-9764-0289a2b32339 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:25.659627 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:25.739212 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79d5b366-690a-43be-b472-b91612095b00 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:25.770562 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73211c6a-aab0-4b99-9764-0289a2b32339 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:25.784118 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a83a122b-94dc-4bdc-a5c0-194ea9058737 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:25.785400 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83a122b-94dc-4bdc-a5c0-194ea9058737 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:25.785883 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83a122b-94dc-4bdc-a5c0-194ea9058737 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:25.786351 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83a122b-94dc-4bdc-a5c0-194ea9058737 None admin] PATH_INFO: `/v3/services` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:25.787467 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a83a122b-94dc-4bdc-a5c0-194ea9058737 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:25.787982 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a83a122b-94dc-4bdc-a5c0-194ea9058737 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:25.790475 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a83a122b-94dc-4bdc-a5c0-194ea9058737 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:25.795272 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:25.800080 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 467/947] 10.222.0.27 () {56 vars in 1122 bytes} [Sun Sep 11 00:40:25 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:25.800080 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c69910c-fb48-4945-8a2d-e52d22941e84 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:25.800080 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c69910c-fb48-4945-8a2d-e52d22941e84 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:25.800080 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c69910c-fb48-4945-8a2d-e52d22941e84 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:25.800080 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 468/948] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:40:25 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:40:25.804811 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c69910c-fb48-4945-8a2d-e52d22941e84 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:25.813879 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3adeef7-b3db-45fd-9769-b10636bd3a31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:25.814539 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3adeef7-b3db-45fd-9769-b10636bd3a31 None admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:25.814841 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3adeef7-b3db-45fd-9769-b10636bd3a31 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:25.815116 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3adeef7-b3db-45fd-9769-b10636bd3a31 None admin] PATH_INFO: `/v3/projects/b504a5058bc6474c8cc077dfeec16799/users/9fc4274836844ae4ad5d16d74c36f671/roles/a6b5856c8f5c4144b39cab1b2916ac93` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:25.817298 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3adeef7-b3db-45fd-9769-b10636bd3a31 None admin] RBAC: Authorizing `identity:create_grant(project_id=b504a5058bc6474c8cc077dfeec16799, user_id=9fc4274836844ae4ad5d16d74c36f671, role_id=a6b5856c8f5c4144b39cab1b2916ac93)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:25.819502 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3adeef7-b3db-45fd-9769-b10636bd3a31 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:25.842819 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 481/949] 10.222.0.27 () {58 vars in 1439 bytes} [Sun Sep 11 00:40:25 2022] PUT /identity/v3/projects/b504a5058bc6474c8cc077dfeec16799/users/9fc4274836844ae4ad5d16d74c36f671/roles/a6b5856c8f5c4144b39cab1b2916ac93 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:40:25.842819 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef5763d-fc49-46c2-873f-60f49a2e7302 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:25.842819 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef5763d-fc49-46c2-873f-60f49a2e7302 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:25.842819 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef5763d-fc49-46c2-873f-60f49a2e7302 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:25.861401 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-fef5763d-fc49-46c2-873f-60f49a2e7302 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:40:25.862737 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-fef5763d-fc49-46c2-873f-60f49a2e7302 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:40:25.946557 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c69910c-fb48-4945-8a2d-e52d22941e84 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:25.949455 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:25.959369 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 482/950] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:40:25 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:40:25.992144 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92433022-e40f-4495-a55c-9e99136dab12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:25.992675 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92433022-e40f-4495-a55c-9e99136dab12 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:25.992950 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92433022-e40f-4495-a55c-9e99136dab12 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:25.993220 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92433022-e40f-4495-a55c-9e99136dab12 None admin] PATH_INFO: `/v3/services` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:25.993861 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92433022-e40f-4495-a55c-9e99136dab12 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:25.994165 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92433022-e40f-4495-a55c-9e99136dab12 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:25.995595 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92433022-e40f-4495-a55c-9e99136dab12 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:26.001679 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 469/951] 10.222.0.27 () {56 vars in 1122 bytes} [Sun Sep 11 00:40:25 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:26.008428 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fef5763d-fc49-46c2-873f-60f49a2e7302 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:26.132044 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fef5763d-fc49-46c2-873f-60f49a2e7302 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:26.149092 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:26.173383 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df1e4fda-4238-41a5-8438-0f622d50fcd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:26.174350 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df1e4fda-4238-41a5-8438-0f622d50fcd4 None admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:26.174872 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df1e4fda-4238-41a5-8438-0f622d50fcd4 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:26.175383 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df1e4fda-4238-41a5-8438-0f622d50fcd4 None admin] PATH_INFO: `/v3/endpoints` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:26.176491 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df1e4fda-4238-41a5-8438-0f622d50fcd4 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:26.178232 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df1e4fda-4238-41a5-8438-0f622d50fcd4 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:26.201667 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 483/952] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:40:26 2022] POST /identity/v3/endpoints => generated 349 bytes in 194 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:40:26.244996 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ee8758-6507-43e6-9b9f-06fcb9dd8a39 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:26.245198 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ee8758-6507-43e6-9b9f-06fcb9dd8a39 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:26.245449 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ee8758-6507-43e6-9b9f-06fcb9dd8a39 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:26.247006 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 470/953] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:40:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:40:26.252897 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e04bfa46-aaa7-4126-acc2-d2021bf9cc33 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:26.253454 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e04bfa46-aaa7-4126-acc2-d2021bf9cc33 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:26.253943 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e04bfa46-aaa7-4126-acc2-d2021bf9cc33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:26.255188 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ee8758-6507-43e6-9b9f-06fcb9dd8a39 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:26.274639 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-e04bfa46-aaa7-4126-acc2-d2021bf9cc33 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:40:26.275979 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-e04bfa46-aaa7-4126-acc2-d2021bf9cc33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:40:26.386882 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 484/954] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:40:26 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:40:26.398565 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ee8758-6507-43e6-9b9f-06fcb9dd8a39 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:26.437029 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c0b2e28-834f-4ad1-996f-99a2eb2d8383 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:26.437656 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0b2e28-834f-4ad1-996f-99a2eb2d8383 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:26.437931 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0b2e28-834f-4ad1-996f-99a2eb2d8383 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:26.438197 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0b2e28-834f-4ad1-996f-99a2eb2d8383 None admin] PATH_INFO: `/v3/services/compute` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:26.438777 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c0b2e28-834f-4ad1-996f-99a2eb2d8383 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:26.440177 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c0b2e28-834f-4ad1-996f-99a2eb2d8383 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:26.449753 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-3c0b2e28-834f-4ad1-996f-99a2eb2d8383 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 11 00:40:26.449753 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:26.449753 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 11 00:40:26.449753 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 11 00:40:26.449753 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 11 00:40:26.449753 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 11 00:40:26.449753 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 11 00:40:26.449753 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 11 00:40:26.449753 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 11 00:40:26.449753 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:26.449753 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 11 00:40:26.449753 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:26.449753 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:26.449753 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:26.449753 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:26.449753 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:26.449753 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:26.450859 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:26.450859 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:26.450859 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:26.450859 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:26.450859 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:26.450859 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:26.450859 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 11 00:40:26.450859 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 11 00:40:26.450859 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 11 00:40:26.450859 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 11 00:40:26.450859 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:26.450859 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:26.450859 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:26.450859 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:26.450859 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:26.450859 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:26.451740 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:26.451740 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:26.451740 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:26.451740 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:26.451740 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:26.451740 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:26.451740 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:26.451740 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:26.451740 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:26.451740 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:26.451740 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 11 00:40:26.451740 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 11 00:40:26.451740 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 11 00:40:26.451740 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:26.451740 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 471/955] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:40:26 2022] GET /identity/v3/services/compute => generated 88 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:40:26.459344 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e04bfa46-aaa7-4126-acc2-d2021bf9cc33 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:26.561706 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:26.568306 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e04bfa46-aaa7-4126-acc2-d2021bf9cc33 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:26.611029 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1339e4b-016a-4169-aff7-16daa2f09523 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:26.611643 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1339e4b-016a-4169-aff7-16daa2f09523 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:26.611923 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1339e4b-016a-4169-aff7-16daa2f09523 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:26.612193 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1339e4b-016a-4169-aff7-16daa2f09523 None admin] PATH_INFO: `/v3/services` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:26.612829 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1339e4b-016a-4169-aff7-16daa2f09523 None admin] RBAC: Adding query filter params (name=compute) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:26.613129 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1339e4b-016a-4169-aff7-16daa2f09523 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:26.614505 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1339e4b-016a-4169-aff7-16daa2f09523 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:26.620467 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 485/956] 10.222.0.27 () {56 vars in 1112 bytes} [Sun Sep 11 00:40:26 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:26.626159 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c0b2e28-834f-4ad1-996f-99a2eb2d8383 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:26.663222 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:26.688004 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1339e4b-016a-4169-aff7-16daa2f09523 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:26.742460 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c0b2e28-834f-4ad1-996f-99a2eb2d8383 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:26.790342 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e35a3055-0c30-41f2-8226-1c7ed9a0a012 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:26.791164 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35a3055-0c30-41f2-8226-1c7ed9a0a012 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:26.791452 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35a3055-0c30-41f2-8226-1c7ed9a0a012 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:26.791731 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35a3055-0c30-41f2-8226-1c7ed9a0a012 None admin] PATH_INFO: `/v3/services` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:26.792374 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e35a3055-0c30-41f2-8226-1c7ed9a0a012 None admin] RBAC: Adding query filter params (type=compute) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:26.792678 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e35a3055-0c30-41f2-8226-1c7ed9a0a012 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:26.794272 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e35a3055-0c30-41f2-8226-1c7ed9a0a012 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:26.797926 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1339e4b-016a-4169-aff7-16daa2f09523 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:26.799670 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 472/957] 10.222.0.27 () {56 vars in 1112 bytes} [Sun Sep 11 00:40:26 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:26.839917 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec8309e7-2f68-45d5-a8c8-e2c688c4935b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:26.840516 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8309e7-2f68-45d5-a8c8-e2c688c4935b None admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:26.840793 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8309e7-2f68-45d5-a8c8-e2c688c4935b None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:26.841116 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8309e7-2f68-45d5-a8c8-e2c688c4935b None admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:26.841807 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec8309e7-2f68-45d5-a8c8-e2c688c4935b None admin] RBAC: Authorizing `identity:create_project()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:26.843410 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec8309e7-2f68-45d5-a8c8-e2c688c4935b None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:26.846299 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: WARNING keystone.server.flask.common [None req-ec8309e7-2f68-45d5-a8c8-e2c688c4935b None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Sep 11 00:40:26.869705 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.sql.core [None req-ec8309e7-2f68-45d5-a8c8-e2c688c4935b None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 11 00:40:26.869705 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Sep 11 00:40:26.869705 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [parameters: {'id': '6b6980e553bb40bba1ade9c6040b003e', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Sep 11 00:40:26.869705 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=87924) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 11 00:40:26.889082 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-ec8309e7-2f68-45d5-a8c8-e2c688c4935b None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 6b6980e553bb40bba1ade9c6040b003e.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 6b6980e553bb40bba1ade9c6040b003e. Sep 11 00:40:26.889082 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:26.889082 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 11 00:40:26.889082 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 11 00:40:26.889082 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 11 00:40:26.889082 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 11 00:40:26.889082 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 11 00:40:26.889082 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application result = self._query(query) Sep 11 00:40:26.889082 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 11 00:40:26.889082 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application conn.query(q) Sep 11 00:40:26.889082 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 11 00:40:26.889082 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 11 00:40:26.889082 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 11 00:40:26.889990 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application result.read() Sep 11 00:40:26.889990 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 11 00:40:26.889990 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 11 00:40:26.889990 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 11 00:40:26.889990 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 11 00:40:26.889990 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 11 00:40:26.889990 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 11 00:40:26.889990 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 11 00:40:26.889990 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 11 00:40:26.889990 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 11 00:40:26.889990 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:26.889990 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 11 00:40:26.889990 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:26.889990 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:26.889990 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 11 00:40:26.889990 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 11 00:40:26.889990 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Sep 11 00:40:26.889990 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return project_ref.to_dict() Sep 11 00:40:26.891162 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 11 00:40:26.891162 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application next(self.gen) Sep 11 00:40:26.891162 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 11 00:40:26.891162 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application yield resource Sep 11 00:40:26.891162 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 11 00:40:26.891162 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application next(self.gen) Sep 11 00:40:26.891162 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 11 00:40:26.891162 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application self.session.rollback() Sep 11 00:40:26.891162 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 11 00:40:26.891162 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application self.force_reraise() Sep 11 00:40:26.891162 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 11 00:40:26.891162 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise self.value Sep 11 00:40:26.891162 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 11 00:40:26.891162 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 11 00:40:26.891162 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 11 00:40:26.891162 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application session.commit() Sep 11 00:40:26.891162 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 11 00:40:26.892081 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 11 00:40:26.892081 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 11 00:40:26.892081 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application self._prepare_impl() Sep 11 00:40:26.892081 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 11 00:40:26.892081 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application self.session.flush() Sep 11 00:40:26.892081 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 11 00:40:26.892081 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application self._flush(objects) Sep 11 00:40:26.892081 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 11 00:40:26.892081 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 11 00:40:26.892081 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 11 00:40:26.892081 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application compat.raise_( Sep 11 00:40:26.892081 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 11 00:40:26.892081 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception Sep 11 00:40:26.892081 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 11 00:40:26.892081 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application flush_context.execute() Sep 11 00:40:26.892081 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 11 00:40:26.892081 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rec.execute(self) Sep 11 00:40:26.893001 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 11 00:40:26.893001 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 11 00:40:26.893001 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 11 00:40:26.893001 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 11 00:40:26.893001 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Sep 11 00:40:26.893001 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application c = connection._execute_20( Sep 11 00:40:26.893001 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 11 00:40:26.893001 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 11 00:40:26.893001 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 11 00:40:26.893001 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 11 00:40:26.893001 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 11 00:40:26.893001 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 11 00:40:26.893001 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 11 00:40:26.893001 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 11 00:40:26.893001 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 11 00:40:26.893814 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 11 00:40:26.893814 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 11 00:40:26.893814 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception Sep 11 00:40:26.893814 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 11 00:40:26.893814 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 11 00:40:26.893814 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 11 00:40:26.893814 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 11 00:40:26.893814 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 11 00:40:26.893814 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application result = self._query(query) Sep 11 00:40:26.893814 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 11 00:40:26.893814 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application conn.query(q) Sep 11 00:40:26.893814 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 11 00:40:26.893814 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 11 00:40:26.893814 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 11 00:40:26.893814 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application result.read() Sep 11 00:40:26.893814 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 11 00:40:26.893814 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 11 00:40:26.894752 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 11 00:40:26.894752 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 11 00:40:26.894752 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 11 00:40:26.894752 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 11 00:40:26.894752 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 11 00:40:26.894752 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 11 00:40:26.894752 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 11 00:40:26.894752 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Sep 11 00:40:26.894752 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [parameters: {'id': '6b6980e553bb40bba1ade9c6040b003e', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Sep 11 00:40:26.894752 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 11 00:40:26.894752 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:26.894752 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 11 00:40:26.894752 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:26.894752 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:26.894752 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Sep 11 00:40:26.897591 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Sep 11 00:40:26.897591 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 11 00:40:26.897591 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 11 00:40:26.897591 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Sep 11 00:40:26.897591 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:26.897591 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 11 00:40:26.897591 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:26.897591 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:26.897591 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:26.897591 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:26.897591 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:26.897591 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:26.897591 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:26.897591 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:26.897591 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:26.897591 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:26.897591 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:26.898735 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:26.898735 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Sep 11 00:40:26.898735 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Sep 11 00:40:26.898735 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:26.898735 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:26.898735 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Sep 11 00:40:26.898735 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.Conflict( Sep 11 00:40:26.898735 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 6b6980e553bb40bba1ade9c6040b003e. Sep 11 00:40:26.898735 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:26.898735 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 486/958] 10.222.0.27 () {60 vars in 1140 bytes} [Sun Sep 11 00:40:26 2022] POST /identity/v3/projects => generated 262 bytes in 203 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:40:26.899455 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e35a3055-0c30-41f2-8226-1c7ed9a0a012 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:27.029590 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e35a3055-0c30-41f2-8226-1c7ed9a0a012 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:27.058651 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:27.078819 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d877a1ff-2970-4a5f-aac2-73a93ee35025 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:27.078819 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d877a1ff-2970-4a5f-aac2-73a93ee35025 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:27.078819 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d877a1ff-2970-4a5f-aac2-73a93ee35025 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:27.078819 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d877a1ff-2970-4a5f-aac2-73a93ee35025 None admin] PATH_INFO: `/v3/projects/service` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:27.095773 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d877a1ff-2970-4a5f-aac2-73a93ee35025 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:27.097189 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d877a1ff-2970-4a5f-aac2-73a93ee35025 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:27.110293 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-d877a1ff-2970-4a5f-aac2-73a93ee35025 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:40:27.110293 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:27.110293 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:27.110293 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:27.110293 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:27.110293 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:27.110293 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:27.110293 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:27.110293 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:27.110293 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:27.110293 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:27.110293 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:27.110293 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 11 00:40:27.110293 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 11 00:40:27.110293 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 11 00:40:27.111296 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 11 00:40:27.111296 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:27.111296 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:27.111296 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:27.111296 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:27.111296 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:27.111296 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:27.111296 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:27.111296 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:27.111296 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:27.111296 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:27.111296 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:27.111296 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:27.111296 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:27.111296 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:27.111296 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:27.111296 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:27.112244 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 11 00:40:27.112244 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 11 00:40:27.112244 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 11 00:40:27.112244 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 11 00:40:27.112244 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 11 00:40:27.112244 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 11 00:40:27.112244 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:40:27.112244 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:27.118006 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 473/959] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:40:26 2022] GET /identity/v3/projects/service => generated 88 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:27.124122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec8309e7-2f68-45d5-a8c8-e2c688c4935b None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:27.161081 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:27.265794 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec8309e7-2f68-45d5-a8c8-e2c688c4935b None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:27.305858 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e97ac1dc-20b8-4852-8dc7-4f10707a006d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:27.306475 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97ac1dc-20b8-4852-8dc7-4f10707a006d None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:27.306757 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97ac1dc-20b8-4852-8dc7-4f10707a006d None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:27.307059 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97ac1dc-20b8-4852-8dc7-4f10707a006d None admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:27.307728 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e97ac1dc-20b8-4852-8dc7-4f10707a006d None admin] RBAC: Adding query filter params (name=service) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:27.308031 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e97ac1dc-20b8-4852-8dc7-4f10707a006d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:27.309676 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e97ac1dc-20b8-4852-8dc7-4f10707a006d None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:27.340846 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 487/960] 10.222.0.27 () {56 vars in 1112 bytes} [Sun Sep 11 00:40:27 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:28.258918 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae900d25-7244-406a-ad7e-bc68b6e89656 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:28.258918 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae900d25-7244-406a-ad7e-bc68b6e89656 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:28.258918 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae900d25-7244-406a-ad7e-bc68b6e89656 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:28.260022 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 474/961] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:40:28 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:40:28.265629 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3697df45-e18d-4c73-9e2b-f9af4c428a60 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:28.266432 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3697df45-e18d-4c73-9e2b-f9af4c428a60 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:28.266935 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3697df45-e18d-4c73-9e2b-f9af4c428a60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:28.287587 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-3697df45-e18d-4c73-9e2b-f9af4c428a60 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:40:28.289593 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-3697df45-e18d-4c73-9e2b-f9af4c428a60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:40:28.363883 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 488/962] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:40:28 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:40:28.516581 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0277e8-e3bf-416f-974f-f41d966941fb None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:28.516909 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0277e8-e3bf-416f-974f-f41d966941fb None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:28.517157 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0277e8-e3bf-416f-974f-f41d966941fb None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:28.518768 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 475/963] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:40:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:40:28.524746 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33bf3477-48ae-4b1e-9016-cd8c6db36b52 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:28.525038 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33bf3477-48ae-4b1e-9016-cd8c6db36b52 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:28.525311 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33bf3477-48ae-4b1e-9016-cd8c6db36b52 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:28.533752 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:28.546086 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-33bf3477-48ae-4b1e-9016-cd8c6db36b52 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:40:28.547312 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-33bf3477-48ae-4b1e-9016-cd8c6db36b52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:40:28.618606 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 489/964] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:40:28 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:40:28.665713 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c0277e8-e3bf-416f-974f-f41d966941fb None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:28.776383 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c0277e8-e3bf-416f-974f-f41d966941fb None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:28.813544 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31085ec7-af62-4f8a-884b-6ece71166fb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:28.814577 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31085ec7-af62-4f8a-884b-6ece71166fb3 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:28.815072 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31085ec7-af62-4f8a-884b-6ece71166fb3 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:28.815551 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31085ec7-af62-4f8a-884b-6ece71166fb3 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:28.827208 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31085ec7-af62-4f8a-884b-6ece71166fb3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:40:28.832304 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-31085ec7-af62-4f8a-884b-6ece71166fb3 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 11 00:40:28.832304 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:28.832304 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:28.832304 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:28.832304 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:28.832304 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:28.832304 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:28.832304 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:28.832304 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:28.832304 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:28.832304 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:28.832304 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:28.832304 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:40:28.832304 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:40:28.832304 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:40:28.832304 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:40:28.833254 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:40:28.833254 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:40:28.833254 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:40:28.833254 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:40:28.833254 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:28.833254 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:28.833254 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:28.833254 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:28.833254 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:28.833254 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:28.833254 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:28.833254 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:28.833254 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:28.833254 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:28.833254 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:28.833254 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:28.835077 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:28.835077 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:28.835077 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:28.835077 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:28.835077 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:40:28.835077 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:40:28.835077 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:40:28.835077 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:40:28.835077 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:40:28.835077 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:40:28.835077 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 11 00:40:28.835077 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:28.837678 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 476/965] 10.222.0.27 () {56 vars in 1120 bytes} [Sun Sep 11 00:40:28 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:28.843771 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33bf3477-48ae-4b1e-9016-cd8c6db36b52 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:28.910470 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31085ec7-af62-4f8a-884b-6ece71166fb3 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:28.931572 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:28.955365 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33bf3477-48ae-4b1e-9016-cd8c6db36b52 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:28.995737 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45cf7e49-fa04-49ba-b3ee-54a7c2dc5375 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:28.997044 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45cf7e49-fa04-49ba-b3ee-54a7c2dc5375 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:28.997556 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45cf7e49-fa04-49ba-b3ee-54a7c2dc5375 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:28.998040 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45cf7e49-fa04-49ba-b3ee-54a7c2dc5375 None admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:28.999548 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45cf7e49-fa04-49ba-b3ee-54a7c2dc5375 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:29.000047 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45cf7e49-fa04-49ba-b3ee-54a7c2dc5375 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:29.001642 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45cf7e49-fa04-49ba-b3ee-54a7c2dc5375 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:29.003340 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:29.013744 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 490/966] 10.222.0.27 () {56 vars in 1115 bytes} [Sun Sep 11 00:40:28 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:29.019897 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45cf7e49-fa04-49ba-b3ee-54a7c2dc5375 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:29.026341 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31085ec7-af62-4f8a-884b-6ece71166fb3 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:29.064044 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa5bba8f-a03b-430d-8801-e28f7cd9d72b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:29.064742 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa5bba8f-a03b-430d-8801-e28f7cd9d72b None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:29.065084 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa5bba8f-a03b-430d-8801-e28f7cd9d72b None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:29.065429 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa5bba8f-a03b-430d-8801-e28f7cd9d72b None admin] PATH_INFO: `/v3/services/object-store` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:29.066432 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa5bba8f-a03b-430d-8801-e28f7cd9d72b None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:29.067540 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa5bba8f-a03b-430d-8801-e28f7cd9d72b None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:29.077187 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-aa5bba8f-a03b-430d-8801-e28f7cd9d72b None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 11 00:40:29.077187 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:29.077187 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 11 00:40:29.077187 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 11 00:40:29.077187 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 11 00:40:29.077187 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 11 00:40:29.077187 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 11 00:40:29.077187 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 11 00:40:29.077187 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 11 00:40:29.077187 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:29.077187 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 11 00:40:29.077187 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:29.077187 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:29.077187 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:29.077187 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:29.077187 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:29.077187 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:29.078318 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:29.078318 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:29.078318 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:29.078318 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:29.078318 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:29.078318 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:29.078318 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 11 00:40:29.078318 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 11 00:40:29.078318 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 11 00:40:29.078318 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 11 00:40:29.078318 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:29.078318 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:29.078318 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:29.078318 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:29.078318 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:29.078318 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:29.079554 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:29.079554 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:29.079554 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:29.079554 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:29.079554 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:29.079554 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:29.079554 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:29.079554 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:29.079554 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:29.079554 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:29.079554 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 11 00:40:29.079554 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 11 00:40:29.079554 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 11 00:40:29.079554 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:29.083437 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 477/967] 10.222.0.27 () {56 vars in 1126 bytes} [Sun Sep 11 00:40:28 2022] GET /identity/v3/services/object-store => generated 93 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:40:29.084874 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa5bba8f-a03b-430d-8801-e28f7cd9d72b None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:29.153934 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45cf7e49-fa04-49ba-b3ee-54a7c2dc5375 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:29.182022 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:29.196831 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a3c6f36-f3bd-4d6a-8e1d-6cc373d8f404 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:29.197685 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3c6f36-f3bd-4d6a-8e1d-6cc373d8f404 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:29.198106 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3c6f36-f3bd-4d6a-8e1d-6cc373d8f404 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:29.198544 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3c6f36-f3bd-4d6a-8e1d-6cc373d8f404 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:29.206429 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa5bba8f-a03b-430d-8801-e28f7cd9d72b None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:29.212109 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a3c6f36-f3bd-4d6a-8e1d-6cc373d8f404 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:29.213739 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a3c6f36-f3bd-4d6a-8e1d-6cc373d8f404 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:29.231387 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-8a3c6f36-f3bd-4d6a-8e1d-6cc373d8f404 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:40:29.231387 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:29.231387 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:29.231387 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:29.231387 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:29.231387 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:29.231387 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:29.231387 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:29.231387 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:29.231387 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:29.231387 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:29.231387 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:29.231387 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:40:29.231387 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:40:29.231387 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:40:29.231387 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:40:29.232414 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:29.232414 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:29.232414 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:29.232414 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:29.232414 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:29.232414 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:29.232414 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:29.232414 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:29.232414 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:29.232414 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:29.232414 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:29.232414 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:29.232414 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:29.232414 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:29.232414 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:29.232414 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:29.232414 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:40:29.233329 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:40:29.233329 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:40:29.233329 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:29.233329 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 491/968] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:40:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:40:29.234394 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db0acd8-feb2-4f5f-b568-d45faf2045fb None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:29.234668 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db0acd8-feb2-4f5f-b568-d45faf2045fb None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:29.234959 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db0acd8-feb2-4f5f-b568-d45faf2045fb None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:29.236281 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 492/969] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:40:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:40:29.240583 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6db0acd8-feb2-4f5f-b568-d45faf2045fb None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:29.248866 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4fe085a-b86e-4951-8ee8-58f848213599 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:29.249474 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4fe085a-b86e-4951-8ee8-58f848213599 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:29.249750 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4fe085a-b86e-4951-8ee8-58f848213599 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:29.250019 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4fe085a-b86e-4951-8ee8-58f848213599 None admin] PATH_INFO: `/v3/services` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:29.250755 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4fe085a-b86e-4951-8ee8-58f848213599 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:29.251127 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4fe085a-b86e-4951-8ee8-58f848213599 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:29.252488 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4fe085a-b86e-4951-8ee8-58f848213599 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:29.257312 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 478/970] 10.222.0.27 () {56 vars in 1122 bytes} [Sun Sep 11 00:40:29 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:29.258480 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efea24df-9027-40a0-8d49-2b3726587d5a None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:29.258762 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efea24df-9027-40a0-8d49-2b3726587d5a None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:29.259068 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efea24df-9027-40a0-8d49-2b3726587d5a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:29.277950 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-efea24df-9027-40a0-8d49-2b3726587d5a None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:40:29.279182 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-efea24df-9027-40a0-8d49-2b3726587d5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:40:29.304635 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:29.347711 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 479/971] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:40:29 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:40:29.348647 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efea24df-9027-40a0-8d49-2b3726587d5a None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:29.375159 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6db0acd8-feb2-4f5f-b568-d45faf2045fb None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:29.414241 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a4ac14-e189-4d3b-abfe-b0e34cfa8146 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:29.414865 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a4ac14-e189-4d3b-abfe-b0e34cfa8146 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:29.415146 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a4ac14-e189-4d3b-abfe-b0e34cfa8146 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:29.415420 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a4ac14-e189-4d3b-abfe-b0e34cfa8146 None admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:29.416073 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a4ac14-e189-4d3b-abfe-b0e34cfa8146 None admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:29.416431 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a4ac14-e189-4d3b-abfe-b0e34cfa8146 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:29.417807 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a4ac14-e189-4d3b-abfe-b0e34cfa8146 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:29.432131 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 493/972] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:40:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:29.439809 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a4ac14-e189-4d3b-abfe-b0e34cfa8146 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:29.444385 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:29.481093 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efea24df-9027-40a0-8d49-2b3726587d5a None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:29.527184 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d59a3e-f316-4ffc-9c2f-651a8b6bd8cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:29.527790 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d59a3e-f316-4ffc-9c2f-651a8b6bd8cd None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:29.528065 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d59a3e-f316-4ffc-9c2f-651a8b6bd8cd None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:29.528333 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d59a3e-f316-4ffc-9c2f-651a8b6bd8cd None admin] PATH_INFO: `/v3/services` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:29.528960 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14d59a3e-f316-4ffc-9c2f-651a8b6bd8cd None admin] RBAC: Adding query filter params (type=object-store) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:29.529257 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14d59a3e-f316-4ffc-9c2f-651a8b6bd8cd None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:29.530898 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14d59a3e-f316-4ffc-9c2f-651a8b6bd8cd None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:29.543018 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 480/973] 10.222.0.27 () {56 vars in 1122 bytes} [Sun Sep 11 00:40:29 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:29.544367 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d59a3e-f316-4ffc-9c2f-651a8b6bd8cd None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:29.577582 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a4ac14-e189-4d3b-abfe-b0e34cfa8146 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:29.591890 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:29.630811 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5c8bd39-af46-4864-b245-f2aeccf929e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:29.631464 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c8bd39-af46-4864-b245-f2aeccf929e9 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:29.631738 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c8bd39-af46-4864-b245-f2aeccf929e9 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:29.632002 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c8bd39-af46-4864-b245-f2aeccf929e9 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:29.639402 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5c8bd39-af46-4864-b245-f2aeccf929e9 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:29.640839 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5c8bd39-af46-4864-b245-f2aeccf929e9 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:29.653231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-a5c8bd39-af46-4864-b245-f2aeccf929e9 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 11 00:40:29.653231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:29.653231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:29.653231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:29.653231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:29.653231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:29.653231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:29.653231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:29.653231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:29.653231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:29.653231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:29.653231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:29.653231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 11 00:40:29.653231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 11 00:40:29.653231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 11 00:40:29.653231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 11 00:40:29.654231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:29.654231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:29.654231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 11 00:40:29.654231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:40:29.654231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 11 00:40:29.654231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:40:29.654231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:29.654231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:29.654231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:29.654231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:29.654231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:29.654231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:29.654231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:29.654231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:29.654231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:29.654231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:29.654231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:29.654231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:29.655639 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:29.655639 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:29.655639 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 11 00:40:29.655639 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 11 00:40:29.655639 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 11 00:40:29.655639 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 11 00:40:29.655639 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 11 00:40:29.655639 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 11 00:40:29.655639 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 11 00:40:29.655639 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:29.655639 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 494/974] 10.222.0.27 () {56 vars in 1117 bytes} [Sun Sep 11 00:40:29 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:29.655639 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5c8bd39-af46-4864-b245-f2aeccf929e9 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:29.679084 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d59a3e-f316-4ffc-9c2f-651a8b6bd8cd None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:29.716207 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b1deba5-14cd-450b-a03c-fff97501526c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:29.716814 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1deba5-14cd-450b-a03c-fff97501526c None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:29.717087 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1deba5-14cd-450b-a03c-fff97501526c None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:29.717358 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1deba5-14cd-450b-a03c-fff97501526c None admin] PATH_INFO: `/v3/endpoints` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:29.718057 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b1deba5-14cd-450b-a03c-fff97501526c None admin] RBAC: Adding query filter params (interface=admin, service_id=b4c75091a7f04c45ae258fb24687ccc0, region_id=RegionOne) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:29.718360 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b1deba5-14cd-450b-a03c-fff97501526c None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:29.719743 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b1deba5-14cd-450b-a03c-fff97501526c None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:29.725524 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 481/975] 10.222.0.27 () {56 vars in 1249 bytes} [Sun Sep 11 00:40:29 2022] GET /identity/v3/endpoints?service_id=b4c75091a7f04c45ae258fb24687ccc0&interface=admin®ion_id=RegionOne => generated 193 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:29.726328 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b1deba5-14cd-450b-a03c-fff97501526c None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:29.728790 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:29.764572 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5c8bd39-af46-4864-b245-f2aeccf929e9 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:29.798848 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4372bde5-07b3-4b37-ab7e-ecfc0cce132b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:29.799448 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4372bde5-07b3-4b37-ab7e-ecfc0cce132b None admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:29.799727 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4372bde5-07b3-4b37-ab7e-ecfc0cce132b None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:29.799998 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4372bde5-07b3-4b37-ab7e-ecfc0cce132b None admin] PATH_INFO: `/v3/services` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:29.800644 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4372bde5-07b3-4b37-ab7e-ecfc0cce132b None admin] RBAC: Authorizing `identity:create_service()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:29.802018 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4372bde5-07b3-4b37-ab7e-ecfc0cce132b None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:29.816411 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 495/976] 10.222.0.27 () {60 vars in 1141 bytes} [Sun Sep 11 00:40:29 2022] POST /identity/v3/services => generated 242 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:40:29.817584 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538055ba-8706-4a0d-b66c-dba7083a4266 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:29.818412 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538055ba-8706-4a0d-b66c-dba7083a4266 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:29.818679 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538055ba-8706-4a0d-b66c-dba7083a4266 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:29.824180 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 496/977] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:40:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:40:29.824180 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-538055ba-8706-4a0d-b66c-dba7083a4266 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:29.836692 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b1deba5-14cd-450b-a03c-fff97501526c None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:29.855105 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:29.888294 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1b55caf-c954-43bd-a5c9-2f950846ac59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:29.888943 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1b55caf-c954-43bd-a5c9-2f950846ac59 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:29.889199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1b55caf-c954-43bd-a5c9-2f950846ac59 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:29.889470 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1b55caf-c954-43bd-a5c9-2f950846ac59 None admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:29.890205 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1b55caf-c954-43bd-a5c9-2f950846ac59 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:29.890532 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1b55caf-c954-43bd-a5c9-2f950846ac59 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:29.892326 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1b55caf-c954-43bd-a5c9-2f950846ac59 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:29.919122 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 482/978] 10.222.0.27 () {56 vars in 1149 bytes} [Sun Sep 11 00:40:29 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:29.920517 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63fe3cd-17d7-4806-8b2c-af9b01dd4c73 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:29.920793 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63fe3cd-17d7-4806-8b2c-af9b01dd4c73 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:29.921058 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63fe3cd-17d7-4806-8b2c-af9b01dd4c73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:29.939568 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-e63fe3cd-17d7-4806-8b2c-af9b01dd4c73 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:40:29.940776 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-e63fe3cd-17d7-4806-8b2c-af9b01dd4c73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:40:29.950640 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-538055ba-8706-4a0d-b66c-dba7083a4266 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:29.993837 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d298fa4f-91c3-438f-aea5-06ea1dca91b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:29.994384 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d298fa4f-91c3-438f-aea5-06ea1dca91b9 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:29.994661 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d298fa4f-91c3-438f-aea5-06ea1dca91b9 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:29.994970 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d298fa4f-91c3-438f-aea5-06ea1dca91b9 None admin] PATH_INFO: `/v3/services` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:29.995580 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d298fa4f-91c3-438f-aea5-06ea1dca91b9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:29.996957 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d298fa4f-91c3-438f-aea5-06ea1dca91b9 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:29.999690 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:30.004276 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 497/979] 10.222.0.27 () {56 vars in 1087 bytes} [Sun Sep 11 00:40:29 2022] GET /identity/v3/services => generated 1727 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:40:30.008068 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d298fa4f-91c3-438f-aea5-06ea1dca91b9 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:30.022829 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 483/980] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:40:29 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:40:30.129985 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:30.140090 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d298fa4f-91c3-438f-aea5-06ea1dca91b9 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:30.178947 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7245f13d-9dc3-4c38-97bc-148f8ad1382a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:30.179575 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7245f13d-9dc3-4c38-97bc-148f8ad1382a None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:30.179852 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7245f13d-9dc3-4c38-97bc-148f8ad1382a None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:30.180171 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7245f13d-9dc3-4c38-97bc-148f8ad1382a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:30.254886 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7245f13d-9dc3-4c38-97bc-148f8ad1382a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:30.256447 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7245f13d-9dc3-4c38-97bc-148f8ad1382a None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:30.299822 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e63fe3cd-17d7-4806-8b2c-af9b01dd4c73 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:30.361168 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:30.372430 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 498/981] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:40:30 2022] GET /identity/v3/auth/tokens => generated 1570 bytes in 368 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 11 00:40:30.377927 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7245f13d-9dc3-4c38-97bc-148f8ad1382a None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:30.419145 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e63fe3cd-17d7-4806-8b2c-af9b01dd4c73 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:30.455622 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224bc2c5-1bf3-4645-8dcc-f0f3a60e5d49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:30.456233 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224bc2c5-1bf3-4645-8dcc-f0f3a60e5d49 None admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:30.456515 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224bc2c5-1bf3-4645-8dcc-f0f3a60e5d49 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:30.456787 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224bc2c5-1bf3-4645-8dcc-f0f3a60e5d49 None admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:30.457434 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-224bc2c5-1bf3-4645-8dcc-f0f3a60e5d49 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:30.459473 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-224bc2c5-1bf3-4645-8dcc-f0f3a60e5d49 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:30.462062 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: WARNING keystone.server.flask.common [None req-224bc2c5-1bf3-4645-8dcc-f0f3a60e5d49 None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Sep 11 00:40:30.476460 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.sql.core [None req-224bc2c5-1bf3-4645-8dcc-f0f3a60e5d49 None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 11 00:40:30.476460 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Sep 11 00:40:30.476460 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [parameters: {'user_id': '7b9aaa3dadd14e1ea5993cb0e2b113b6', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Sep 11 00:40:30.476460 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=87923) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 11 00:40:30.484195 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-224bc2c5-1bf3-4645-8dcc-f0f3a60e5d49 None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Sep 11 00:40:30.484195 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:30.484195 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 11 00:40:30.484195 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 11 00:40:30.484195 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 11 00:40:30.484195 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 11 00:40:30.484195 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 11 00:40:30.484195 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application result = self._query(query) Sep 11 00:40:30.484195 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 11 00:40:30.484195 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application conn.query(q) Sep 11 00:40:30.484195 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 11 00:40:30.484195 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 11 00:40:30.484195 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 11 00:40:30.484195 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application result.read() Sep 11 00:40:30.484195 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 11 00:40:30.485312 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 11 00:40:30.485312 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 11 00:40:30.485312 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 11 00:40:30.485312 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 11 00:40:30.485312 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 11 00:40:30.485312 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 11 00:40:30.485312 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 11 00:40:30.485312 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 11 00:40:30.485312 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:30.485312 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 11 00:40:30.485312 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:30.485312 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:30.485312 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 11 00:40:30.485312 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 11 00:40:30.485312 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Sep 11 00:40:30.485312 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Sep 11 00:40:30.485312 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 11 00:40:30.485312 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application next(self.gen) Sep 11 00:40:30.486645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 11 00:40:30.486645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application yield resource Sep 11 00:40:30.486645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 11 00:40:30.486645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application next(self.gen) Sep 11 00:40:30.486645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 11 00:40:30.486645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application self.session.rollback() Sep 11 00:40:30.486645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 11 00:40:30.486645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application self.force_reraise() Sep 11 00:40:30.486645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 11 00:40:30.486645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise self.value Sep 11 00:40:30.486645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 11 00:40:30.486645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 11 00:40:30.486645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 11 00:40:30.486645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application session.commit() Sep 11 00:40:30.486645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 11 00:40:30.486645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 11 00:40:30.486645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 11 00:40:30.487823 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application self._prepare_impl() Sep 11 00:40:30.487823 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 11 00:40:30.487823 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application self.session.flush() Sep 11 00:40:30.487823 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 11 00:40:30.487823 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application self._flush(objects) Sep 11 00:40:30.487823 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 11 00:40:30.487823 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 11 00:40:30.487823 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 11 00:40:30.487823 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application compat.raise_( Sep 11 00:40:30.487823 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 11 00:40:30.487823 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception Sep 11 00:40:30.487823 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 11 00:40:30.487823 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application flush_context.execute() Sep 11 00:40:30.487823 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 11 00:40:30.487823 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rec.execute(self) Sep 11 00:40:30.487823 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 11 00:40:30.487823 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 11 00:40:30.488896 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 11 00:40:30.488896 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 11 00:40:30.488896 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Sep 11 00:40:30.488896 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application result = connection._execute_20( Sep 11 00:40:30.488896 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 11 00:40:30.488896 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 11 00:40:30.488896 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 11 00:40:30.488896 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 11 00:40:30.488896 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 11 00:40:30.488896 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 11 00:40:30.488896 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 11 00:40:30.488896 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 11 00:40:30.488896 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 11 00:40:30.488896 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 11 00:40:30.488896 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 11 00:40:30.488896 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception Sep 11 00:40:30.490477 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 11 00:40:30.490477 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 11 00:40:30.490477 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 11 00:40:30.490477 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 11 00:40:30.490477 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 11 00:40:30.490477 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application result = self._query(query) Sep 11 00:40:30.490477 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 11 00:40:30.490477 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application conn.query(q) Sep 11 00:40:30.490477 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 11 00:40:30.490477 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 11 00:40:30.490477 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 11 00:40:30.490477 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application result.read() Sep 11 00:40:30.490477 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 11 00:40:30.490477 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 11 00:40:30.490477 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 11 00:40:30.490477 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 11 00:40:30.497054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 11 00:40:30.497054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 11 00:40:30.497054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 11 00:40:30.497054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 11 00:40:30.497054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 11 00:40:30.497054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Sep 11 00:40:30.497054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [parameters: {'user_id': '7b9aaa3dadd14e1ea5993cb0e2b113b6', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Sep 11 00:40:30.497054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 11 00:40:30.497054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:30.497054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 11 00:40:30.497054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:30.497054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:30.497054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:30.497054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:30.497054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:30.498960 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:30.498960 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:30.498960 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:30.498960 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:30.498960 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:30.498960 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:30.498960 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:30.498960 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Sep 11 00:40:30.498960 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Sep 11 00:40:30.498960 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:30.498960 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:30.498960 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 11 00:40:30.498960 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:40:30.498960 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 11 00:40:30.498960 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:40:30.498960 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Sep 11 00:40:30.498960 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Sep 11 00:40:30.501178 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Sep 11 00:40:30.501178 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Sep 11 00:40:30.501178 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 11 00:40:30.501178 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 11 00:40:30.501178 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Sep 11 00:40:30.501178 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:30.501178 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 484/982] 10.222.0.27 () {60 vars in 1131 bytes} [Sun Sep 11 00:40:30 2022] POST /identity/v3/users => generated 160 bytes in 186 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:40:30.502063 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7245f13d-9dc3-4c38-97bc-148f8ad1382a None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:30.504597 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224bc2c5-1bf3-4645-8dcc-f0f3a60e5d49 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:30.529045 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197ef791-484e-4e78-996a-982f3d312c7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:30.530195 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197ef791-484e-4e78-996a-982f3d312c7f None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:30.530634 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197ef791-484e-4e78-996a-982f3d312c7f None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:30.531083 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197ef791-484e-4e78-996a-982f3d312c7f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:30.544304 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-197ef791-484e-4e78-996a-982f3d312c7f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:30.546064 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-197ef791-484e-4e78-996a-982f3d312c7f None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:30.565104 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-197ef791-484e-4e78-996a-982f3d312c7f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:40:30.565104 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:30.565104 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:30.565104 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:30.565104 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:30.565104 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:30.565104 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:30.565104 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:30.565104 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:30.565104 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:30.565104 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:30.565104 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:30.565104 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:40:30.565104 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:40:30.565104 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:40:30.565104 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:40:30.566136 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:30.566136 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:30.566136 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:30.566136 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:30.566136 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:30.566136 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:30.566136 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:30.566136 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:30.566136 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:30.566136 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:30.566136 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:30.566136 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:30.566136 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:30.566136 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:30.566136 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:30.566136 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:30.566136 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:40:30.567195 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:40:30.567195 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:40:30.567195 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:30.567368 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:30.574352 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 499/983] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:40:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:40:30.575152 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197ef791-484e-4e78-996a-982f3d312c7f None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:30.628993 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224bc2c5-1bf3-4645-8dcc-f0f3a60e5d49 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:30.632053 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:30.668627 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d97b2d8f-4b2b-409a-9ac1-2c71f9380b82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:30.669677 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97b2d8f-4b2b-409a-9ac1-2c71f9380b82 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:30.670183 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97b2d8f-4b2b-409a-9ac1-2c71f9380b82 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:30.670669 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97b2d8f-4b2b-409a-9ac1-2c71f9380b82 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:30.679396 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d97b2d8f-4b2b-409a-9ac1-2c71f9380b82 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:30.681619 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d97b2d8f-4b2b-409a-9ac1-2c71f9380b82 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:30.695393 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-d97b2d8f-4b2b-409a-9ac1-2c71f9380b82 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 11 00:40:30.695393 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:30.695393 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:30.695393 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:30.695393 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:30.695393 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:30.695393 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:30.695393 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:30.695393 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:30.695393 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:30.695393 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:30.695393 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:30.695393 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 11 00:40:30.695393 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 11 00:40:30.695393 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 11 00:40:30.695393 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 11 00:40:30.696383 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:30.696383 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:30.696383 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 11 00:40:30.696383 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:40:30.696383 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 11 00:40:30.696383 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:40:30.696383 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:30.696383 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:30.696383 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:30.696383 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:30.696383 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:30.696383 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:30.696383 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:30.696383 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:30.696383 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:30.696383 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:30.696383 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:30.696383 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:30.697375 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:30.697375 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:30.697375 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 11 00:40:30.697375 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 11 00:40:30.697375 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 11 00:40:30.697375 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 11 00:40:30.697375 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 11 00:40:30.697375 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 11 00:40:30.697375 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 11 00:40:30.697375 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:30.697940 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197ef791-484e-4e78-996a-982f3d312c7f None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:30.700147 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 485/984] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:40:30 2022] GET /identity/v3/users/cinder => generated 84 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:40:30.707501 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d97b2d8f-4b2b-409a-9ac1-2c71f9380b82 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:30.737969 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03049ded-6579-4dd0-ba7f-24298b3cf1eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:30.739038 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03049ded-6579-4dd0-ba7f-24298b3cf1eb None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:30.739508 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03049ded-6579-4dd0-ba7f-24298b3cf1eb None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:30.739997 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03049ded-6579-4dd0-ba7f-24298b3cf1eb None admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:30.740852 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03049ded-6579-4dd0-ba7f-24298b3cf1eb None admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:30.741354 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03049ded-6579-4dd0-ba7f-24298b3cf1eb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:30.743473 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03049ded-6579-4dd0-ba7f-24298b3cf1eb None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:30.763286 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 500/985] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:40:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:40:30.764484 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03049ded-6579-4dd0-ba7f-24298b3cf1eb None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:30.846490 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d97b2d8f-4b2b-409a-9ac1-2c71f9380b82 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:30.858959 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:30.889448 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c471ad-4c91-47d2-84a2-0b266b862b71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:30.890045 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c471ad-4c91-47d2-84a2-0b266b862b71 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:30.890319 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c471ad-4c91-47d2-84a2-0b266b862b71 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:30.890585 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c471ad-4c91-47d2-84a2-0b266b862b71 None admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:30.891483 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8c471ad-4c91-47d2-84a2-0b266b862b71 None admin] RBAC: Adding query filter params (name=cinder) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:30.891789 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8c471ad-4c91-47d2-84a2-0b266b862b71 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:30.893110 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03049ded-6579-4dd0-ba7f-24298b3cf1eb None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:30.893235 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8c471ad-4c91-47d2-84a2-0b266b862b71 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:30.896900 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:30.934667 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d6a6af-b7e9-4c51-b0b9-f7ed54d3f3a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:30.935743 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d6a6af-b7e9-4c51-b0b9-f7ed54d3f3a6 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:30.936255 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d6a6af-b7e9-4c51-b0b9-f7ed54d3f3a6 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:30.936862 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d6a6af-b7e9-4c51-b0b9-f7ed54d3f3a6 None admin] PATH_INFO: `/v3/projects/service` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:30.938951 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 486/986] 10.222.0.27 () {56 vars in 1101 bytes} [Sun Sep 11 00:40:30 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:30.951082 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75d6a6af-b7e9-4c51-b0b9-f7ed54d3f3a6 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:30.953086 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75d6a6af-b7e9-4c51-b0b9-f7ed54d3f3a6 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:30.967210 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-75d6a6af-b7e9-4c51-b0b9-f7ed54d3f3a6 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:40:30.967210 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:30.967210 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:30.967210 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:30.967210 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:30.967210 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:30.967210 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:30.967210 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:30.967210 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:30.967210 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:30.967210 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:30.967210 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:30.967210 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 11 00:40:30.967210 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 11 00:40:30.967210 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 11 00:40:30.968184 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 11 00:40:30.968184 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:30.968184 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:30.968184 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:30.968184 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:30.968184 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:30.968184 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:30.968184 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:30.968184 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:30.968184 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:30.968184 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:30.968184 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:30.968184 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:30.968184 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:30.968184 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:30.968184 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:30.968184 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:30.969122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 11 00:40:30.969122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 11 00:40:30.969122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 11 00:40:30.969122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 11 00:40:30.969122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 11 00:40:30.969122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 11 00:40:30.969122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:40:30.969122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:30.971743 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 501/987] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:40:30 2022] GET /identity/v3/projects/service => generated 88 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:30.978434 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c471ad-4c91-47d2-84a2-0b266b862b71 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:31.097680 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c471ad-4c91-47d2-84a2-0b266b862b71 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:31.139843 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713a6778-a420-48c3-aea6-b7dc4afb58a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:31.140840 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713a6778-a420-48c3-aea6-b7dc4afb58a7 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:31.141119 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713a6778-a420-48c3-aea6-b7dc4afb58a7 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:31.141390 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713a6778-a420-48c3-aea6-b7dc4afb58a7 None admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:31.142083 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-713a6778-a420-48c3-aea6-b7dc4afb58a7 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:31.142386 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-713a6778-a420-48c3-aea6-b7dc4afb58a7 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:31.143913 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-713a6778-a420-48c3-aea6-b7dc4afb58a7 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:31.156887 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 487/988] 10.222.0.27 () {56 vars in 1148 bytes} [Sun Sep 11 00:40:30 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:31.163627 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d6a6af-b7e9-4c51-b0b9-f7ed54d3f3a6 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:31.270725 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d6a6af-b7e9-4c51-b0b9-f7ed54d3f3a6 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:31.305430 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15b2716d-b69c-47a0-b96d-988f8f1d0f3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:31.306033 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b2716d-b69c-47a0-b96d-988f8f1d0f3e None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:31.306310 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b2716d-b69c-47a0-b96d-988f8f1d0f3e None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:31.306576 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b2716d-b69c-47a0-b96d-988f8f1d0f3e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:31.307338 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15b2716d-b69c-47a0-b96d-988f8f1d0f3e None admin] RBAC: Adding query filter params (role.id=a6b5856c8f5c4144b39cab1b2916ac93, scope.project.id=b504a5058bc6474c8cc077dfeec16799, user.id=9fc4274836844ae4ad5d16d74c36f671) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:31.307643 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15b2716d-b69c-47a0-b96d-988f8f1d0f3e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:31.309063 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15b2716d-b69c-47a0-b96d-988f8f1d0f3e None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:31.314905 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 502/989] 10.222.0.27 () {56 vars in 1374 bytes} [Sun Sep 11 00:40:31 2022] GET /identity/v3/role_assignments?user.id=9fc4274836844ae4ad5d16d74c36f671&scope.project.id=b504a5058bc6474c8cc077dfeec16799&role.id=a6b5856c8f5c4144b39cab1b2916ac93 => generated 608 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:32.132675 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f095da-e108-4528-873b-063be972e8c9 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:32.133206 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f095da-e108-4528-873b-063be972e8c9 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:32.133206 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f095da-e108-4528-873b-063be972e8c9 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:32.134589 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 488/990] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:40:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:40:32.140381 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b255ac-03e5-4531-ab99-12e362a8d4d9 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:32.140805 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b255ac-03e5-4531-ab99-12e362a8d4d9 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:32.140989 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b255ac-03e5-4531-ab99-12e362a8d4d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:32.159427 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-99b255ac-03e5-4531-ab99-12e362a8d4d9 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:40:32.160905 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-99b255ac-03e5-4531-ab99-12e362a8d4d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:40:32.188367 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:32.238373 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 503/991] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:40:32 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:40:32.291241 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1aa8f7-bdb1-49e0-b97f-7a1d1c9ac008 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:32.291542 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1aa8f7-bdb1-49e0-b97f-7a1d1c9ac008 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:32.291801 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1aa8f7-bdb1-49e0-b97f-7a1d1c9ac008 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:32.293268 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 489/992] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:40:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:40:32.298465 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31bc0ba-30ea-4503-99d5-6e2cda445d64 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:32.298750 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31bc0ba-30ea-4503-99d5-6e2cda445d64 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:32.299050 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31bc0ba-30ea-4503-99d5-6e2cda445d64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:32.317793 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-d31bc0ba-30ea-4503-99d5-6e2cda445d64 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:40:32.318997 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-d31bc0ba-30ea-4503-99d5-6e2cda445d64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:40:32.384442 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 504/993] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:40:32 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:40:32.504648 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f1aa8f7-bdb1-49e0-b97f-7a1d1c9ac008 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:32.533521 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:32.610775 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f1aa8f7-bdb1-49e0-b97f-7a1d1c9ac008 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:32.645810 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ba1d89-4e3b-4900-af16-05211a5005b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:32.646603 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ba1d89-4e3b-4900-af16-05211a5005b0 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:32.646903 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ba1d89-4e3b-4900-af16-05211a5005b0 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:32.647174 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ba1d89-4e3b-4900-af16-05211a5005b0 None admin] PATH_INFO: `/v3/services/compute` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:32.647750 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0ba1d89-4e3b-4900-af16-05211a5005b0 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:32.649108 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0ba1d89-4e3b-4900-af16-05211a5005b0 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:32.660679 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d31bc0ba-30ea-4503-99d5-6e2cda445d64 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:32.660841 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-b0ba1d89-4e3b-4900-af16-05211a5005b0 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 11 00:40:32.660841 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:32.660841 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 11 00:40:32.660841 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 11 00:40:32.660841 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 11 00:40:32.660841 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 11 00:40:32.660841 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 11 00:40:32.660841 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 11 00:40:32.660841 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 11 00:40:32.660841 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:32.660841 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 11 00:40:32.660841 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:32.660841 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:32.660841 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:32.660841 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:32.660841 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:32.660841 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:32.662022 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:32.662022 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:32.662022 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:32.662022 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:32.662022 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:32.662022 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:32.662022 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 11 00:40:32.662022 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 11 00:40:32.662022 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 11 00:40:32.662022 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 11 00:40:32.662022 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:32.662022 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:32.662022 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:32.662022 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:32.662022 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:32.662022 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:32.663148 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:32.663148 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:32.663148 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:32.663148 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:32.663148 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:32.663148 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:32.663148 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:32.663148 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:32.663148 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:32.663148 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:32.663148 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 11 00:40:32.663148 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 11 00:40:32.663148 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 11 00:40:32.663148 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:32.663148 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 490/994] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:40:32 2022] GET /identity/v3/services/compute => generated 88 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:32.666741 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ba1d89-4e3b-4900-af16-05211a5005b0 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:32.747888 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:32.761060 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d31bc0ba-30ea-4503-99d5-6e2cda445d64 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:32.767921 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:32.780642 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ba1d89-4e3b-4900-af16-05211a5005b0 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:32.798592 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd2534a-5bdc-4722-87bd-0d15363a96ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:32.799330 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd2534a-5bdc-4722-87bd-0d15363a96ad None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:32.799613 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd2534a-5bdc-4722-87bd-0d15363a96ad None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:32.799886 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd2534a-5bdc-4722-87bd-0d15363a96ad None admin] PATH_INFO: `/v3/services/object-store` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:32.800467 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fd2534a-5bdc-4722-87bd-0d15363a96ad None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:32.801821 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fd2534a-5bdc-4722-87bd-0d15363a96ad None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:32.811501 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-2fd2534a-5bdc-4722-87bd-0d15363a96ad None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 11 00:40:32.811501 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:32.811501 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 11 00:40:32.811501 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 11 00:40:32.811501 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 11 00:40:32.811501 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 11 00:40:32.811501 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 11 00:40:32.811501 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 11 00:40:32.811501 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 11 00:40:32.811501 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:32.811501 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 11 00:40:32.811501 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:32.811501 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:32.811501 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:32.811501 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:32.811501 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:32.811501 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:32.812524 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:32.812524 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:32.812524 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:32.812524 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:32.812524 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:32.812524 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:32.812524 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 11 00:40:32.812524 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 11 00:40:32.812524 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 11 00:40:32.812524 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 11 00:40:32.812524 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:32.812524 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:32.812524 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:32.812524 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:32.812524 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:32.812524 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:32.813370 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:32.813370 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:32.813370 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:32.813370 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:32.813370 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:32.813370 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:32.813370 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:32.813370 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:32.813370 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:32.813370 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:32.813370 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 11 00:40:32.813370 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 11 00:40:32.813370 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 11 00:40:32.813370 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:32.817652 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 505/995] 10.222.0.27 () {56 vars in 1126 bytes} [Sun Sep 11 00:40:32 2022] GET /identity/v3/services/object-store => generated 93 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:40:32.818981 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd2534a-5bdc-4722-87bd-0d15363a96ad None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:32.819881 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9cc9712-96a4-49eb-b1c9-6d693d6c4fa1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:32.820477 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9cc9712-96a4-49eb-b1c9-6d693d6c4fa1 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:32.820751 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9cc9712-96a4-49eb-b1c9-6d693d6c4fa1 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:32.821019 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9cc9712-96a4-49eb-b1c9-6d693d6c4fa1 None admin] PATH_INFO: `/v3/services` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:32.821799 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9cc9712-96a4-49eb-b1c9-6d693d6c4fa1 None admin] RBAC: Adding query filter params (name=compute) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:32.822884 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9cc9712-96a4-49eb-b1c9-6d693d6c4fa1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:32.824400 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9cc9712-96a4-49eb-b1c9-6d693d6c4fa1 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:32.829861 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 491/996] 10.222.0.27 () {56 vars in 1112 bytes} [Sun Sep 11 00:40:32 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:40:32.835560 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9cc9712-96a4-49eb-b1c9-6d693d6c4fa1 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:32.925837 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd2534a-5bdc-4722-87bd-0d15363a96ad None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:32.935993 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9cc9712-96a4-49eb-b1c9-6d693d6c4fa1 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:32.964708 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8add03d9-c61b-4403-a7b7-7c278babf745 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:32.965572 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8add03d9-c61b-4403-a7b7-7c278babf745 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:32.965999 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8add03d9-c61b-4403-a7b7-7c278babf745 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:32.966666 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8add03d9-c61b-4403-a7b7-7c278babf745 None admin] PATH_INFO: `/v3/services` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:32.967879 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8add03d9-c61b-4403-a7b7-7c278babf745 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:32.968570 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8add03d9-c61b-4403-a7b7-7c278babf745 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:32.970138 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8add03d9-c61b-4403-a7b7-7c278babf745 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:32.972108 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74ace73a-a2b5-4ec5-9cd2-8d6432415766 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:32.972717 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ace73a-a2b5-4ec5-9cd2-8d6432415766 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:32.973000 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ace73a-a2b5-4ec5-9cd2-8d6432415766 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:32.973271 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ace73a-a2b5-4ec5-9cd2-8d6432415766 None admin] PATH_INFO: `/v3/services` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:32.974423 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74ace73a-a2b5-4ec5-9cd2-8d6432415766 None admin] RBAC: Adding query filter params (type=compute) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:32.974719 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74ace73a-a2b5-4ec5-9cd2-8d6432415766 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:32.976271 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74ace73a-a2b5-4ec5-9cd2-8d6432415766 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:32.981511 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 492/997] 10.222.0.27 () {56 vars in 1112 bytes} [Sun Sep 11 00:40:32 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:40:32.981999 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 506/998] 10.222.0.27 () {56 vars in 1122 bytes} [Sun Sep 11 00:40:32 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:40:32.982486 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74ace73a-a2b5-4ec5-9cd2-8d6432415766 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:32.986715 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8add03d9-c61b-4403-a7b7-7c278babf745 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:33.004464 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:33.010687 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:33.096249 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74ace73a-a2b5-4ec5-9cd2-8d6432415766 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:33.111446 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8add03d9-c61b-4403-a7b7-7c278babf745 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:33.136524 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b091a68-b5f9-4fe2-8c05-7423d33e6aa7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:33.137357 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b091a68-b5f9-4fe2-8c05-7423d33e6aa7 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:33.137707 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b091a68-b5f9-4fe2-8c05-7423d33e6aa7 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:33.138049 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b091a68-b5f9-4fe2-8c05-7423d33e6aa7 None admin] PATH_INFO: `/v3/services` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:33.138757 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b091a68-b5f9-4fe2-8c05-7423d33e6aa7 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:33.139160 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b091a68-b5f9-4fe2-8c05-7423d33e6aa7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:33.140727 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b091a68-b5f9-4fe2-8c05-7423d33e6aa7 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:33.145936 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 493/999] 10.222.0.27 () {56 vars in 1122 bytes} [Sun Sep 11 00:40:32 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:33.149182 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c26d1ca9-8e33-4b45-ae9d-87ffa8962f71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:33.149797 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26d1ca9-8e33-4b45-ae9d-87ffa8962f71 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:33.150071 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26d1ca9-8e33-4b45-ae9d-87ffa8962f71 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:33.150513 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26d1ca9-8e33-4b45-ae9d-87ffa8962f71 None admin] PATH_INFO: `/v3/endpoints` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:33.151260 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c26d1ca9-8e33-4b45-ae9d-87ffa8962f71 None admin] RBAC: Adding query filter params (interface=public, service_id=f3bd3de36e1a48e9b09ad0f0452af567, region_id=RegionOne) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:33.151567 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c26d1ca9-8e33-4b45-ae9d-87ffa8962f71 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:33.151750 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b091a68-b5f9-4fe2-8c05-7423d33e6aa7 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:33.152923 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c26d1ca9-8e33-4b45-ae9d-87ffa8962f71 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:33.164280 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 507/1000] 10.222.0.27 () {56 vars in 1251 bytes} [Sun Sep 11 00:40:32 2022] GET /identity/v3/endpoints?service_id=f3bd3de36e1a48e9b09ad0f0452af567&interface=public®ion_id=RegionOne => generated 194 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:33.165159 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c26d1ca9-8e33-4b45-ae9d-87ffa8962f71 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:33.237370 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:33.257626 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b091a68-b5f9-4fe2-8c05-7423d33e6aa7 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:33.261650 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:33.283965 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c26d1ca9-8e33-4b45-ae9d-87ffa8962f71 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:33.292096 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9558491-b9b1-40aa-a937-15a6829c258c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:33.292776 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9558491-b9b1-40aa-a937-15a6829c258c None admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:33.293122 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9558491-b9b1-40aa-a937-15a6829c258c None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:33.293459 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9558491-b9b1-40aa-a937-15a6829c258c None admin] PATH_INFO: `/v3/endpoints` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:33.294199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9558491-b9b1-40aa-a937-15a6829c258c None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:33.295645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9558491-b9b1-40aa-a937-15a6829c258c None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:33.320925 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 494/1001] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:40:33 2022] POST /identity/v3/endpoints => generated 325 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:40:33.331901 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ba1f8a-ffab-4d14-8601-a15b6b3c72cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:33.332517 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ba1f8a-ffab-4d14-8601-a15b6b3c72cc None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:33.332794 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ba1f8a-ffab-4d14-8601-a15b6b3c72cc None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:33.333062 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ba1f8a-ffab-4d14-8601-a15b6b3c72cc None admin] PATH_INFO: `/v3/services` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:33.334305 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ba1f8a-ffab-4d14-8601-a15b6b3c72cc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:33.335880 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ba1f8a-ffab-4d14-8601-a15b6b3c72cc None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:33.342913 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 508/1002] 10.222.0.27 () {56 vars in 1087 bytes} [Sun Sep 11 00:40:33 2022] GET /identity/v3/services => generated 1727 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 11 00:40:33.541208 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c0914a-736e-41ef-a7ab-adae22f695b0 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:33.541401 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c0914a-736e-41ef-a7ab-adae22f695b0 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:33.541681 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c0914a-736e-41ef-a7ab-adae22f695b0 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:33.543421 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 495/1003] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:40:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:40:33.549418 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f020234f-3c7c-4515-afbc-1d73b4fdaace None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:33.549711 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f020234f-3c7c-4515-afbc-1d73b4fdaace None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:33.549989 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f020234f-3c7c-4515-afbc-1d73b4fdaace None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:33.569729 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-f020234f-3c7c-4515-afbc-1d73b4fdaace None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:40:33.571026 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-f020234f-3c7c-4515-afbc-1d73b4fdaace None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:40:33.641024 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 509/1004] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:40:33 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:40:33.913444 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c0914a-736e-41ef-a7ab-adae22f695b0 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:34.015234 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c0914a-736e-41ef-a7ab-adae22f695b0 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:34.050346 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5310b0c1-5e45-4145-b180-9555e0137c8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:34.050966 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5310b0c1-5e45-4145-b180-9555e0137c8c None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:34.051240 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5310b0c1-5e45-4145-b180-9555e0137c8c None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:34.051511 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5310b0c1-5e45-4145-b180-9555e0137c8c None admin] PATH_INFO: `/v3/services/image` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:34.052650 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5310b0c1-5e45-4145-b180-9555e0137c8c None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:34.054146 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5310b0c1-5e45-4145-b180-9555e0137c8c None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:34.063414 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-5310b0c1-5e45-4145-b180-9555e0137c8c None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 11 00:40:34.063414 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:34.063414 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 11 00:40:34.063414 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 11 00:40:34.063414 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 11 00:40:34.063414 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 11 00:40:34.063414 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 11 00:40:34.063414 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 11 00:40:34.063414 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 11 00:40:34.063414 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:34.063414 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 11 00:40:34.063414 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:34.063414 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:34.063414 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:34.063414 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:34.063414 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:34.063414 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:34.064456 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:34.064456 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:34.064456 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:34.064456 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:34.064456 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:34.064456 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:34.064456 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 11 00:40:34.064456 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 11 00:40:34.064456 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 11 00:40:34.064456 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 11 00:40:34.064456 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:34.064456 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:34.064456 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:34.064456 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:34.064456 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:34.064456 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:34.065331 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:34.065331 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:34.065331 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:34.065331 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:34.065331 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:34.065331 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:34.065331 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:34.065331 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:34.065331 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:34.065331 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:34.065331 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 11 00:40:34.065331 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 11 00:40:34.065331 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 11 00:40:34.065331 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:34.066742 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 496/1005] 10.222.0.27 () {56 vars in 1105 bytes} [Sun Sep 11 00:40:33 2022] GET /identity/v3/services/image => generated 86 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:40:34.070470 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f020234f-3c7c-4515-afbc-1d73b4fdaace None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:34.145022 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:34.178325 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f020234f-3c7c-4515-afbc-1d73b4fdaace None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:34.211772 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-909a6579-5c81-432c-91e0-6a39d1b7da38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:34.212377 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909a6579-5c81-432c-91e0-6a39d1b7da38 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:34.212654 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909a6579-5c81-432c-91e0-6a39d1b7da38 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:34.212920 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909a6579-5c81-432c-91e0-6a39d1b7da38 None admin] PATH_INFO: `/v3/services` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:34.213572 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-909a6579-5c81-432c-91e0-6a39d1b7da38 None admin] RBAC: Adding query filter params (name=image) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:34.213871 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-909a6579-5c81-432c-91e0-6a39d1b7da38 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:34.215269 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-909a6579-5c81-432c-91e0-6a39d1b7da38 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:34.220159 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 510/1006] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:40:34 2022] GET /identity/v3/services?name=image => generated 122 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:34.225803 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5310b0c1-5e45-4145-b180-9555e0137c8c None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:34.240503 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:34.340267 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5310b0c1-5e45-4145-b180-9555e0137c8c None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:34.374323 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3ec6d83-3c43-415f-9d61-620f0c7a49de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:34.374947 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3ec6d83-3c43-415f-9d61-620f0c7a49de None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:34.375227 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3ec6d83-3c43-415f-9d61-620f0c7a49de None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:34.375494 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3ec6d83-3c43-415f-9d61-620f0c7a49de None admin] PATH_INFO: `/v3/services` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:34.376117 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3ec6d83-3c43-415f-9d61-620f0c7a49de None admin] RBAC: Adding query filter params (type=image) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:34.376416 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3ec6d83-3c43-415f-9d61-620f0c7a49de None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:34.377772 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3ec6d83-3c43-415f-9d61-620f0c7a49de None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:34.383002 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 497/1007] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:40:34 2022] GET /identity/v3/services?type=image => generated 122 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:35.569721 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e232788b-9070-4816-99eb-9e4fe3cd5f08 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:35.570278 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e232788b-9070-4816-99eb-9e4fe3cd5f08 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:35.570278 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e232788b-9070-4816-99eb-9e4fe3cd5f08 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:35.571297 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f62f99-58eb-4b72-aa49-bbac14ba42d1 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:35.571688 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f62f99-58eb-4b72-aa49-bbac14ba42d1 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:35.571860 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f62f99-58eb-4b72-aa49-bbac14ba42d1 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:35.571957 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 511/1008] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:40:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:40:35.573164 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 498/1009] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:40:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:40:35.577048 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6764fb6b-4aa1-49f5-bbfb-119ad13fab80 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:35.577342 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6764fb6b-4aa1-49f5-bbfb-119ad13fab80 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:35.577615 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6764fb6b-4aa1-49f5-bbfb-119ad13fab80 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:35.578112 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38907d2d-f756-4a0d-81cf-ea455a8d212b None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:35.578390 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38907d2d-f756-4a0d-81cf-ea455a8d212b None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:35.578660 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38907d2d-f756-4a0d-81cf-ea455a8d212b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:35.596272 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-6764fb6b-4aa1-49f5-bbfb-119ad13fab80 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:40:35.597456 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-38907d2d-f756-4a0d-81cf-ea455a8d212b None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:40:35.597568 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-6764fb6b-4aa1-49f5-bbfb-119ad13fab80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:40:35.599717 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-38907d2d-f756-4a0d-81cf-ea455a8d212b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:40:35.663472 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:35.669178 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 512/1011] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:40:35 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:40:35.669324 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 499/1011] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:40:35 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 11 00:40:35.936552 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6764fb6b-4aa1-49f5-bbfb-119ad13fab80 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:35.939439 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38907d2d-f756-4a0d-81cf-ea455a8d212b None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:36.005632 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:36.041918 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38907d2d-f756-4a0d-81cf-ea455a8d212b None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:36.047689 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6764fb6b-4aa1-49f5-bbfb-119ad13fab80 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:36.076035 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e3e816-7afd-4c5c-806d-c74a8262e8da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:36.076693 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e3e816-7afd-4c5c-806d-c74a8262e8da None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:36.078833 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e3e816-7afd-4c5c-806d-c74a8262e8da None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:36.078833 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e3e816-7afd-4c5c-806d-c74a8262e8da None admin] PATH_INFO: `/v3/domains/default` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:36.078833 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19e3e816-7afd-4c5c-806d-c74a8262e8da None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:36.079786 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19e3e816-7afd-4c5c-806d-c74a8262e8da None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:36.086822 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 500/1012] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:40:35 2022] GET /identity/v3/domains/default => generated 199 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:40:36.087587 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e3e816-7afd-4c5c-806d-c74a8262e8da None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:36.089820 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9669cad9-9463-4510-9f72-140cb666c370 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:36.090461 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9669cad9-9463-4510-9f72-140cb666c370 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:36.090736 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9669cad9-9463-4510-9f72-140cb666c370 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:36.091034 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9669cad9-9463-4510-9f72-140cb666c370 None admin] PATH_INFO: `/v3/services/compute` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:36.091629 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9669cad9-9463-4510-9f72-140cb666c370 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:36.092991 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9669cad9-9463-4510-9f72-140cb666c370 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:36.105497 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-9669cad9-9463-4510-9f72-140cb666c370 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 11 00:40:36.105497 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:36.105497 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 11 00:40:36.105497 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 11 00:40:36.105497 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 11 00:40:36.105497 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 11 00:40:36.105497 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 11 00:40:36.105497 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 11 00:40:36.105497 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 11 00:40:36.105497 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:36.105497 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 11 00:40:36.105497 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:36.105497 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:36.105497 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:36.105497 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:36.105497 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:36.105497 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:36.106514 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:36.106514 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:36.106514 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:36.106514 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:36.106514 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:36.106514 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:36.106514 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 11 00:40:36.106514 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 11 00:40:36.106514 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 11 00:40:36.106514 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 11 00:40:36.106514 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:36.106514 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:36.106514 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:36.106514 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:36.106514 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:36.106514 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:36.107446 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:36.107446 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:36.107446 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:36.107446 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:36.107446 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:36.107446 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:36.107446 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:36.107446 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:36.107446 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:36.107446 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:36.107446 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 11 00:40:36.107446 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 11 00:40:36.107446 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 11 00:40:36.107446 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:36.107446 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 513/1013] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:40:35 2022] GET /identity/v3/services/compute => generated 88 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:36.112486 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9669cad9-9463-4510-9f72-140cb666c370 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:36.161715 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:36.200927 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e3e816-7afd-4c5c-806d-c74a8262e8da None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:36.222192 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9669cad9-9463-4510-9f72-140cb666c370 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:36.238075 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d5aeb7-6bed-419e-ba2c-cccc07135b0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:36.238671 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d5aeb7-6bed-419e-ba2c-cccc07135b0e None admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:36.238973 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d5aeb7-6bed-419e-ba2c-cccc07135b0e None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:36.239787 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d5aeb7-6bed-419e-ba2c-cccc07135b0e None admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:36.240712 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d5aeb7-6bed-419e-ba2c-cccc07135b0e None admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:36.242275 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d5aeb7-6bed-419e-ba2c-cccc07135b0e None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:36.250768 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:36.261066 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfbd2f5-1d9f-498d-a1c7-56746a04235a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:36.261981 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfbd2f5-1d9f-498d-a1c7-56746a04235a None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:36.262281 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfbd2f5-1d9f-498d-a1c7-56746a04235a None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:36.262612 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfbd2f5-1d9f-498d-a1c7-56746a04235a None admin] PATH_INFO: `/v3/services` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:36.263242 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecfbd2f5-1d9f-498d-a1c7-56746a04235a None admin] RBAC: Adding query filter params (name=compute) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:36.263571 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecfbd2f5-1d9f-498d-a1c7-56746a04235a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:36.264962 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecfbd2f5-1d9f-498d-a1c7-56746a04235a None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:36.270693 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 514/1014] 10.222.0.27 () {56 vars in 1112 bytes} [Sun Sep 11 00:40:36 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:40:36.272982 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 501/1015] 10.222.0.27 () {60 vars in 1141 bytes} [Sun Sep 11 00:40:36 2022] POST /identity/v3/projects => generated 311 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:40:36.277028 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfbd2f5-1d9f-498d-a1c7-56746a04235a None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:36.391022 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfbd2f5-1d9f-498d-a1c7-56746a04235a None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:36.429532 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5d143d-e596-43b5-b4f6-36397fae22d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:36.430500 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5d143d-e596-43b5-b4f6-36397fae22d1 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:36.430999 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5d143d-e596-43b5-b4f6-36397fae22d1 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:36.431483 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5d143d-e596-43b5-b4f6-36397fae22d1 None admin] PATH_INFO: `/v3/services` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:36.432332 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be5d143d-e596-43b5-b4f6-36397fae22d1 None admin] RBAC: Adding query filter params (type=compute) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:36.432830 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be5d143d-e596-43b5-b4f6-36397fae22d1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:36.434527 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be5d143d-e596-43b5-b4f6-36397fae22d1 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:36.445553 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 515/1016] 10.222.0.27 () {56 vars in 1112 bytes} [Sun Sep 11 00:40:36 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:40:36.447438 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d5aeb7-6bed-419e-ba2c-cccc07135b0e None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:36.555413 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d5aeb7-6bed-419e-ba2c-cccc07135b0e None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:36.558448 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321f6685-eec9-483c-b39f-413d707c18fd None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:36.558448 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321f6685-eec9-483c-b39f-413d707c18fd None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:36.558448 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321f6685-eec9-483c-b39f-413d707c18fd None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:36.559791 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 516/1017] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:40:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 11 00:40:36.564639 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3755c2-08ff-4846-84a3-1b634bcb2900 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:36.564955 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3755c2-08ff-4846-84a3-1b634bcb2900 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:36.565419 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3755c2-08ff-4846-84a3-1b634bcb2900 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:36.576957 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:36.584216 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-7f3755c2-08ff-4846-84a3-1b634bcb2900 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:40:36.585400 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-7f3755c2-08ff-4846-84a3-1b634bcb2900 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:40:36.593713 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69e6f8e-747a-4e67-be0e-80a6fba315b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:36.594318 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69e6f8e-747a-4e67-be0e-80a6fba315b2 None admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:36.594592 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69e6f8e-747a-4e67-be0e-80a6fba315b2 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:36.594895 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69e6f8e-747a-4e67-be0e-80a6fba315b2 None admin] PATH_INFO: `/v3/endpoints` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:36.595570 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d69e6f8e-747a-4e67-be0e-80a6fba315b2 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:36.596925 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d69e6f8e-747a-4e67-be0e-80a6fba315b2 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:36.617119 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 502/1018] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:40:36 2022] POST /identity/v3/endpoints => generated 334 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:40:36.648136 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:36.674827 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 517/1019] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:40:36 2022] POST /identity/v3/auth/tokens => generated 1873 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:40:36.948369 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69e6f8e-747a-4e67-be0e-80a6fba315b2 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:37.051576 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69e6f8e-747a-4e67-be0e-80a6fba315b2 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:37.091198 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bbd07b9-3598-4098-8f3c-212849e00fbd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:37.091829 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bbd07b9-3598-4098-8f3c-212849e00fbd None admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:37.092104 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bbd07b9-3598-4098-8f3c-212849e00fbd None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:37.092369 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bbd07b9-3598-4098-8f3c-212849e00fbd None admin] PATH_INFO: `/v3/services` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:37.093053 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bbd07b9-3598-4098-8f3c-212849e00fbd None admin] RBAC: Authorizing `identity:create_service()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:37.094420 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bbd07b9-3598-4098-8f3c-212849e00fbd None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:37.114061 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 503/1020] 10.222.0.27 () {60 vars in 1141 bytes} [Sun Sep 11 00:40:36 2022] POST /identity/v3/services => generated 242 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:40:38.525043 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b9b525e-4c75-45af-8147-bd2dac8aadca None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:38.525591 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b9b525e-4c75-45af-8147-bd2dac8aadca None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:38.525591 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b9b525e-4c75-45af-8147-bd2dac8aadca None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:38.526961 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 518/1021] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:40:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:40:38.532389 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05799e6-bd83-47d4-8625-a6a418c9f554 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:38.532798 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05799e6-bd83-47d4-8625-a6a418c9f554 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:38.532963 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05799e6-bd83-47d4-8625-a6a418c9f554 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:38.551236 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-e05799e6-bd83-47d4-8625-a6a418c9f554 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:40:38.552945 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-e05799e6-bd83-47d4-8625-a6a418c9f554 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:40:38.581388 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:38.628600 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 504/1022] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:40:38 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:40:38.883155 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9506682f-1998-465e-a240-28b87348c09a None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:38.883343 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9506682f-1998-465e-a240-28b87348c09a None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:38.883613 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9506682f-1998-465e-a240-28b87348c09a None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:38.885103 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 519/1023] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:40:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:40:38.890275 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3555a1-d282-432f-9cc8-0c5095f7cf8b None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:38.890567 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3555a1-d282-432f-9cc8-0c5095f7cf8b None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:38.890973 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3555a1-d282-432f-9cc8-0c5095f7cf8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:38.901135 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9506682f-1998-465e-a240-28b87348c09a None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:38.909607 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-ff3555a1-d282-432f-9cc8-0c5095f7cf8b None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:40:38.911645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-ff3555a1-d282-432f-9cc8-0c5095f7cf8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:40:38.981298 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 505/1024] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:40:38 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:40:39.005352 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9506682f-1998-465e-a240-28b87348c09a None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:39.039332 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66632e87-3162-41d1-b6ab-8c2c869bac79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:39.039968 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66632e87-3162-41d1-b6ab-8c2c869bac79 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:39.040246 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66632e87-3162-41d1-b6ab-8c2c869bac79 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:39.040522 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66632e87-3162-41d1-b6ab-8c2c869bac79 None admin] PATH_INFO: `/v3/domains/default` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:39.041636 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66632e87-3162-41d1-b6ab-8c2c869bac79 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:39.043098 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66632e87-3162-41d1-b6ab-8c2c869bac79 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:39.049523 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 520/1025] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:40:38 2022] GET /identity/v3/domains/default => generated 199 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:40:39.055026 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3555a1-d282-432f-9cc8-0c5095f7cf8b None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:39.153431 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:39.164911 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3555a1-d282-432f-9cc8-0c5095f7cf8b None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:39.199142 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c323256-13e7-46ae-94cf-f94fe5b81cb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:39.199772 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c323256-13e7-46ae-94cf-f94fe5b81cb2 None admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:39.200050 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c323256-13e7-46ae-94cf-f94fe5b81cb2 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:39.200318 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c323256-13e7-46ae-94cf-f94fe5b81cb2 None admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:39.200991 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c323256-13e7-46ae-94cf-f94fe5b81cb2 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:39.202488 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c323256-13e7-46ae-94cf-f94fe5b81cb2 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:39.234825 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 506/1026] 10.222.0.27 () {60 vars in 1132 bytes} [Sun Sep 11 00:40:39 2022] POST /identity/v3/users => generated 285 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:40:39.257850 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66632e87-3162-41d1-b6ab-8c2c869bac79 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:39.315084 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:39.334532 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96645cc-5f77-4657-a9d7-68a0d644b3ff None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:39.334851 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96645cc-5f77-4657-a9d7-68a0d644b3ff None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:39.335124 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96645cc-5f77-4657-a9d7-68a0d644b3ff None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:39.336715 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 507/1027] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:40:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:40:39.342265 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309842c8-8094-4700-a58e-1743fcfa0e74 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:39.342566 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309842c8-8094-4700-a58e-1743fcfa0e74 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:39.342846 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309842c8-8094-4700-a58e-1743fcfa0e74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:39.363834 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-309842c8-8094-4700-a58e-1743fcfa0e74 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:40:39.365721 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-309842c8-8094-4700-a58e-1743fcfa0e74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:40:39.376634 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66632e87-3162-41d1-b6ab-8c2c869bac79 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:39.415139 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50b5fbc-b5e4-43df-8e6e-f8c4e2b6f7aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:39.415691 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50b5fbc-b5e4-43df-8e6e-f8c4e2b6f7aa None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:39.415965 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50b5fbc-b5e4-43df-8e6e-f8c4e2b6f7aa None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:39.416231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50b5fbc-b5e4-43df-8e6e-f8c4e2b6f7aa None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:39.428797 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e50b5fbc-b5e4-43df-8e6e-f8c4e2b6f7aa None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:40:39.434660 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-e50b5fbc-b5e4-43df-8e6e-f8c4e2b6f7aa None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 11 00:40:39.434660 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:39.434660 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:39.434660 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:39.434660 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:39.434660 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:39.434660 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:39.434660 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:39.434660 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:39.434660 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:39.434660 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:39.434660 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:39.434660 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:40:39.434660 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:40:39.434660 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:40:39.434660 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:40:39.435686 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:40:39.435686 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:40:39.435686 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:40:39.435686 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:40:39.435686 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:39.435686 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:39.435686 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:39.435686 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:39.435686 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:39.435686 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:39.435686 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:39.435686 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:39.435686 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:39.435686 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:39.435686 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:39.435686 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:39.436541 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:39.436541 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:39.436541 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:39.436541 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:39.436541 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:40:39.436541 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:40:39.436541 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:40:39.436541 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:40:39.436541 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:40:39.436541 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:40:39.436541 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 11 00:40:39.436541 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:39.436541 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 521/1028] 10.222.0.27 () {56 vars in 1120 bytes} [Sun Sep 11 00:40:39 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:40:39.442025 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50b5fbc-b5e4-43df-8e6e-f8c4e2b6f7aa None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:39.448739 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 508/1029] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:40:39 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 11 00:40:39.550958 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50b5fbc-b5e4-43df-8e6e-f8c4e2b6f7aa None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:39.563144 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:39.588089 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f927548-196b-4a2f-a9b6-f9f7ba116d49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:39.588920 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f927548-196b-4a2f-a9b6-f9f7ba116d49 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:39.589378 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f927548-196b-4a2f-a9b6-f9f7ba116d49 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:39.589814 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f927548-196b-4a2f-a9b6-f9f7ba116d49 None admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:39.590639 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f927548-196b-4a2f-a9b6-f9f7ba116d49 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:39.591119 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f927548-196b-4a2f-a9b6-f9f7ba116d49 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:39.592634 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f927548-196b-4a2f-a9b6-f9f7ba116d49 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:39.601987 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 522/1030] 10.222.0.27 () {56 vars in 1115 bytes} [Sun Sep 11 00:40:39 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:40:39.607655 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-309842c8-8094-4700-a58e-1743fcfa0e74 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:39.691257 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:39.717869 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f927548-196b-4a2f-a9b6-f9f7ba116d49 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:39.721065 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-309842c8-8094-4700-a58e-1743fcfa0e74 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:39.755759 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b6d704-491e-4950-be58-4de9491f7536 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:39.756396 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b6d704-491e-4950-be58-4de9491f7536 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:39.756670 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b6d704-491e-4950-be58-4de9491f7536 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:39.756940 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b6d704-491e-4950-be58-4de9491f7536 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:39.767094 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4b6d704-491e-4950-be58-4de9491f7536 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:39.768534 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4b6d704-491e-4950-be58-4de9491f7536 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:39.783666 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-c4b6d704-491e-4950-be58-4de9491f7536 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:40:39.783666 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:39.783666 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:39.783666 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:39.783666 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:39.783666 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:39.783666 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:39.783666 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:39.783666 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:39.783666 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:39.783666 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:39.783666 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:39.783666 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:40:39.783666 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:40:39.783666 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:40:39.783666 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:40:39.784689 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:39.784689 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:39.784689 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:39.784689 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:39.784689 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:39.784689 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:39.784689 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:39.784689 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:39.784689 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:39.784689 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:39.784689 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:39.784689 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:39.784689 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:39.784689 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:39.784689 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:39.784689 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:39.784689 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:40:39.785944 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:40:39.785944 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:40:39.785944 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:39.785944 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 509/1031] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:40:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:40:39.791521 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b6d704-491e-4950-be58-4de9491f7536 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:39.833675 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f927548-196b-4a2f-a9b6-f9f7ba116d49 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:39.872972 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c60da8-f4fc-4cc3-a82c-85f9deda41c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:39.873910 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c60da8-f4fc-4cc3-a82c-85f9deda41c1 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:39.874206 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c60da8-f4fc-4cc3-a82c-85f9deda41c1 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:39.874628 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c60da8-f4fc-4cc3-a82c-85f9deda41c1 None admin] PATH_INFO: `/v3/services/image` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:39.875363 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6c60da8-f4fc-4cc3-a82c-85f9deda41c1 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:39.877265 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6c60da8-f4fc-4cc3-a82c-85f9deda41c1 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:39.889192 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-d6c60da8-f4fc-4cc3-a82c-85f9deda41c1 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 11 00:40:39.889192 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:39.889192 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 11 00:40:39.889192 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 11 00:40:39.889192 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 11 00:40:39.889192 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 11 00:40:39.889192 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 11 00:40:39.889192 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 11 00:40:39.889192 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 11 00:40:39.889192 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:39.889192 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 11 00:40:39.889192 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:39.889192 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:39.889192 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:39.889192 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:39.889192 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:39.889192 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:39.890215 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:39.890215 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:39.890215 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:39.890215 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:39.890215 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:39.890215 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:39.890215 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 11 00:40:39.890215 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 11 00:40:39.890215 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 11 00:40:39.890215 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 11 00:40:39.890215 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:39.890215 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:39.890215 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:39.890215 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:39.890215 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:39.890215 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:39.895235 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:39.895235 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:39.895235 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:39.895235 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:39.895235 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:39.895235 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:39.895235 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:39.895235 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:39.895235 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:39.895235 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:39.895235 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 11 00:40:39.895235 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 11 00:40:39.895235 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 11 00:40:39.895235 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:39.895235 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 523/1032] 10.222.0.27 () {56 vars in 1105 bytes} [Sun Sep 11 00:40:39 2022] GET /identity/v3/services/image => generated 86 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:40:39.896811 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b6d704-491e-4950-be58-4de9491f7536 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:39.898279 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c60da8-f4fc-4cc3-a82c-85f9deda41c1 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:39.927613 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:39.934010 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-432c0f9d-2b95-4ddb-8d0a-c46b2cecd0f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:39.934613 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432c0f9d-2b95-4ddb-8d0a-c46b2cecd0f4 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:39.934920 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432c0f9d-2b95-4ddb-8d0a-c46b2cecd0f4 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:39.935191 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432c0f9d-2b95-4ddb-8d0a-c46b2cecd0f4 None admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:39.935834 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-432c0f9d-2b95-4ddb-8d0a-c46b2cecd0f4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:39.936134 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-432c0f9d-2b95-4ddb-8d0a-c46b2cecd0f4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:39.937496 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-432c0f9d-2b95-4ddb-8d0a-c46b2cecd0f4 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:39.941363 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:39.950036 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 510/1033] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:40:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:40:39.955709 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-432c0f9d-2b95-4ddb-8d0a-c46b2cecd0f4 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:40.014223 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c60da8-f4fc-4cc3-a82c-85f9deda41c1 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:40.049322 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c107cdd-7bad-483e-90b7-9afe81a6791a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:40.049931 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c107cdd-7bad-483e-90b7-9afe81a6791a None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:40.050219 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c107cdd-7bad-483e-90b7-9afe81a6791a None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:40.050482 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c107cdd-7bad-483e-90b7-9afe81a6791a None admin] PATH_INFO: `/v3/services` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:40.051140 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c107cdd-7bad-483e-90b7-9afe81a6791a None admin] RBAC: Adding query filter params (name=image) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:40.051451 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c107cdd-7bad-483e-90b7-9afe81a6791a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:40.053424 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c107cdd-7bad-483e-90b7-9afe81a6791a None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:40.062844 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 524/1034] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:40:39 2022] GET /identity/v3/services?name=image => generated 122 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:40:40.064073 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c107cdd-7bad-483e-90b7-9afe81a6791a None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:40.065813 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-432c0f9d-2b95-4ddb-8d0a-c46b2cecd0f4 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:40.108451 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a34b9271-67fd-4c08-9cdf-64c9f1ac1df4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:40.109082 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a34b9271-67fd-4c08-9cdf-64c9f1ac1df4 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:40.109359 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a34b9271-67fd-4c08-9cdf-64c9f1ac1df4 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:40.109628 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a34b9271-67fd-4c08-9cdf-64c9f1ac1df4 None admin] PATH_INFO: `/v3/users/nova` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:40.117952 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a34b9271-67fd-4c08-9cdf-64c9f1ac1df4 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:40.119435 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a34b9271-67fd-4c08-9cdf-64c9f1ac1df4 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:40.131044 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-a34b9271-67fd-4c08-9cdf-64c9f1ac1df4 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 11 00:40:40.131044 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:40.131044 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:40.131044 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:40.131044 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:40.131044 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:40.131044 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:40.131044 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:40.131044 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:40.131044 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:40.131044 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:40.131044 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:40.131044 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 11 00:40:40.131044 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 11 00:40:40.131044 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 11 00:40:40.131044 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 11 00:40:40.132103 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:40.132103 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:40.132103 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 11 00:40:40.132103 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:40:40.132103 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 11 00:40:40.132103 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:40:40.132103 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:40.132103 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:40.132103 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:40.132103 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:40.132103 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:40.132103 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:40.132103 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:40.132103 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:40.132103 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:40.132103 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:40.132103 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:40.132103 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:40.133128 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:40.133128 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:40.133128 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 11 00:40:40.133128 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 11 00:40:40.133128 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 11 00:40:40.133128 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 11 00:40:40.133128 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 11 00:40:40.133128 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 11 00:40:40.133128 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 11 00:40:40.133128 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:40.133128 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 511/1035] 10.222.0.27 () {56 vars in 1093 bytes} [Sun Sep 11 00:40:39 2022] GET /identity/v3/users/nova => generated 82 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:40:40.137859 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a34b9271-67fd-4c08-9cdf-64c9f1ac1df4 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:40.179134 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:40.183422 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c107cdd-7bad-483e-90b7-9afe81a6791a None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:40.188010 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:40.227267 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b435729c-58cc-4e80-9c98-e23e7d94e420 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:40.227878 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b435729c-58cc-4e80-9c98-e23e7d94e420 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:40.228155 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b435729c-58cc-4e80-9c98-e23e7d94e420 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:40.228425 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b435729c-58cc-4e80-9c98-e23e7d94e420 None admin] PATH_INFO: `/v3/services` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:40.229059 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b435729c-58cc-4e80-9c98-e23e7d94e420 None admin] RBAC: Adding query filter params (type=image) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:40.229358 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b435729c-58cc-4e80-9c98-e23e7d94e420 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:40.230936 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b435729c-58cc-4e80-9c98-e23e7d94e420 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:40.236688 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 525/1036] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:40:40 2022] GET /identity/v3/services?type=image => generated 350 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:40:40.242493 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a34b9271-67fd-4c08-9cdf-64c9f1ac1df4 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:40.242839 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b435729c-58cc-4e80-9c98-e23e7d94e420 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:40.278156 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebdf11e8-bf1d-4af8-a287-f0c38e280bc7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:40.278757 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebdf11e8-bf1d-4af8-a287-f0c38e280bc7 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:40.279061 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebdf11e8-bf1d-4af8-a287-f0c38e280bc7 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:40.279329 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebdf11e8-bf1d-4af8-a287-f0c38e280bc7 None admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:40.280045 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebdf11e8-bf1d-4af8-a287-f0c38e280bc7 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:40.280348 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebdf11e8-bf1d-4af8-a287-f0c38e280bc7 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:40.281782 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebdf11e8-bf1d-4af8-a287-f0c38e280bc7 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:40.306167 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 512/1037] 10.222.0.27 () {56 vars in 1133 bytes} [Sun Sep 11 00:40:40 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:40:40.307216 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebdf11e8-bf1d-4af8-a287-f0c38e280bc7 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:40.355810 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b435729c-58cc-4e80-9c98-e23e7d94e420 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:40.392866 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-748ccf26-cba7-4578-8b0d-0f41bfa0a2a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:40.393471 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748ccf26-cba7-4578-8b0d-0f41bfa0a2a0 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:40.393754 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748ccf26-cba7-4578-8b0d-0f41bfa0a2a0 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:40.394022 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748ccf26-cba7-4578-8b0d-0f41bfa0a2a0 None admin] PATH_INFO: `/v3/endpoints` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:40.394718 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-748ccf26-cba7-4578-8b0d-0f41bfa0a2a0 None admin] RBAC: Adding query filter params (interface=public, service_id=abfba60f929d4748b46d3c3dc898f8fc, region_id=RegionOne) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:40.395051 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-748ccf26-cba7-4578-8b0d-0f41bfa0a2a0 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:40.396522 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-748ccf26-cba7-4578-8b0d-0f41bfa0a2a0 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:40.401772 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 526/1038] 10.222.0.27 () {56 vars in 1251 bytes} [Sun Sep 11 00:40:40 2022] GET /identity/v3/endpoints?service_id=abfba60f929d4748b46d3c3dc898f8fc&interface=public®ion_id=RegionOne => generated 194 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:40:40.407560 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-748ccf26-cba7-4578-8b0d-0f41bfa0a2a0 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:40.415174 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebdf11e8-bf1d-4af8-a287-f0c38e280bc7 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:40.425351 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:40.439857 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:40.455085 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d73552-4105-42bc-ac80-dbcd6a72df3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:40.455692 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d73552-4105-42bc-ac80-dbcd6a72df3c None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:40.455965 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d73552-4105-42bc-ac80-dbcd6a72df3c None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:40.456234 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d73552-4105-42bc-ac80-dbcd6a72df3c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:40.520926 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-748ccf26-cba7-4578-8b0d-0f41bfa0a2a0 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:40.528256 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12d73552-4105-42bc-ac80-dbcd6a72df3c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:40.529714 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12d73552-4105-42bc-ac80-dbcd6a72df3c None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:40.559807 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-685b3327-4604-4bbe-a561-357c5962338c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:40.560476 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685b3327-4604-4bbe-a561-357c5962338c None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:40.560767 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685b3327-4604-4bbe-a561-357c5962338c None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:40.560963 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685b3327-4604-4bbe-a561-357c5962338c None admin] PATH_INFO: `/v3/services` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:40.561614 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-685b3327-4604-4bbe-a561-357c5962338c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:40.562962 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-685b3327-4604-4bbe-a561-357c5962338c None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:40.582868 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 527/1039] 10.222.0.27 () {56 vars in 1087 bytes} [Sun Sep 11 00:40:40 2022] GET /identity/v3/services => generated 1957 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:40:40.652715 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 513/1040] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:40:40 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 346 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 11 00:40:40.658640 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-685b3327-4604-4bbe-a561-357c5962338c None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:40.760040 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:40.769226 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-685b3327-4604-4bbe-a561-357c5962338c None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:40.804047 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e72602-888b-4887-ac1e-9cc67fffc9e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:40.804587 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e72602-888b-4887-ac1e-9cc67fffc9e7 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:40.804862 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e72602-888b-4887-ac1e-9cc67fffc9e7 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:40.805127 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e72602-888b-4887-ac1e-9cc67fffc9e7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:40.815368 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e72602-888b-4887-ac1e-9cc67fffc9e7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:40.816792 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e72602-888b-4887-ac1e-9cc67fffc9e7 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:40.831122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-12e72602-888b-4887-ac1e-9cc67fffc9e7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:40:40.831122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:40.831122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:40.831122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:40.831122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:40.831122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:40.831122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:40.831122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:40.831122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:40.831122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:40.831122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:40.831122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:40.831122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:40:40.831122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:40:40.831122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:40:40.831122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:40:40.832104 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:40.832104 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:40.832104 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:40.832104 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:40.832104 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:40.832104 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:40.832104 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:40.832104 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:40.832104 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:40.832104 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:40.832104 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:40.832104 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:40.832104 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:40.832104 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:40.832104 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:40.832104 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:40.832104 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:40:40.833075 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:40:40.833075 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:40:40.833075 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:40.833075 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 528/1041] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:40:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:40.838319 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d73552-4105-42bc-ac80-dbcd6a72df3c None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:40.848781 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:40.940284 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d73552-4105-42bc-ac80-dbcd6a72df3c None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:40.975064 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09884f18-9734-40a9-b7c7-6ec590b9c214 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:40.975665 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09884f18-9734-40a9-b7c7-6ec590b9c214 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:40.975937 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09884f18-9734-40a9-b7c7-6ec590b9c214 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:40.976202 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09884f18-9734-40a9-b7c7-6ec590b9c214 None admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:40.976835 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09884f18-9734-40a9-b7c7-6ec590b9c214 None admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:40.977133 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09884f18-9734-40a9-b7c7-6ec590b9c214 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:40.978481 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09884f18-9734-40a9-b7c7-6ec590b9c214 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:40.990353 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 514/1042] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:40:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:40.995777 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e72602-888b-4887-ac1e-9cc67fffc9e7 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:41.101626 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e72602-888b-4887-ac1e-9cc67fffc9e7 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:41.136452 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a21bc986-f121-4b16-b1db-0328d4d37117 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:41.137082 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a21bc986-f121-4b16-b1db-0328d4d37117 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:41.137355 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a21bc986-f121-4b16-b1db-0328d4d37117 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:41.137621 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a21bc986-f121-4b16-b1db-0328d4d37117 None admin] PATH_INFO: `/v3/projects/service` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:41.144265 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a21bc986-f121-4b16-b1db-0328d4d37117 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:41.145826 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a21bc986-f121-4b16-b1db-0328d4d37117 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:41.156873 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-a21bc986-f121-4b16-b1db-0328d4d37117 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:40:41.156873 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:41.156873 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:41.156873 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:41.156873 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:41.156873 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:41.156873 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:41.156873 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:41.156873 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:41.156873 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:41.156873 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:41.156873 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:41.156873 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 11 00:40:41.156873 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 11 00:40:41.156873 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 11 00:40:41.157778 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 11 00:40:41.157778 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:41.157778 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:41.157778 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:41.157778 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:41.157778 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:41.157778 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:41.157778 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:41.157778 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:41.157778 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:41.157778 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:41.157778 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:41.157778 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:41.157778 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:41.157778 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:41.157778 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:41.157778 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:41.158701 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 11 00:40:41.158701 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 11 00:40:41.158701 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 11 00:40:41.158701 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 11 00:40:41.158701 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 11 00:40:41.158701 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 11 00:40:41.158701 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:40:41.158701 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:41.160089 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 529/1043] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:40:40 2022] GET /identity/v3/projects/service => generated 88 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:41.165029 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09884f18-9734-40a9-b7c7-6ec590b9c214 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:41.234859 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:41.271366 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09884f18-9734-40a9-b7c7-6ec590b9c214 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:41.305179 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07102696-6aaa-421d-8ef7-bf3fc3acaf4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:41.305791 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07102696-6aaa-421d-8ef7-bf3fc3acaf4b None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:41.306075 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07102696-6aaa-421d-8ef7-bf3fc3acaf4b None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:41.306348 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07102696-6aaa-421d-8ef7-bf3fc3acaf4b None admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:41.307066 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07102696-6aaa-421d-8ef7-bf3fc3acaf4b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:41.307372 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07102696-6aaa-421d-8ef7-bf3fc3acaf4b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:41.308817 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07102696-6aaa-421d-8ef7-bf3fc3acaf4b None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:41.326883 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 515/1044] 10.222.0.27 () {56 vars in 1148 bytes} [Sun Sep 11 00:40:41 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:41.328088 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a21bc986-f121-4b16-b1db-0328d4d37117 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:41.331928 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:41.435497 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a21bc986-f121-4b16-b1db-0328d4d37117 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:41.469129 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2380ccdb-daa2-420d-ae73-ea49e21bb1c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:41.469759 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2380ccdb-daa2-420d-ae73-ea49e21bb1c4 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:41.470089 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2380ccdb-daa2-420d-ae73-ea49e21bb1c4 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:41.470338 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2380ccdb-daa2-420d-ae73-ea49e21bb1c4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:41.471120 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2380ccdb-daa2-420d-ae73-ea49e21bb1c4 None admin] RBAC: Adding query filter params (role.id=a6b5856c8f5c4144b39cab1b2916ac93, scope.project.id=b504a5058bc6474c8cc077dfeec16799, user.id=835def0e6c6d4365a2767051509eb0d6) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:41.471433 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2380ccdb-daa2-420d-ae73-ea49e21bb1c4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:41.472919 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2380ccdb-daa2-420d-ae73-ea49e21bb1c4 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:41.486827 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 530/1045] 10.222.0.27 () {56 vars in 1374 bytes} [Sun Sep 11 00:40:41 2022] GET /identity/v3/role_assignments?user.id=835def0e6c6d4365a2767051509eb0d6&scope.project.id=b504a5058bc6474c8cc077dfeec16799&role.id=a6b5856c8f5c4144b39cab1b2916ac93 => generated 259 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:40:41.507831 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca69d51-81a1-4612-b1d4-2112c4e3f510 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:41.508118 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca69d51-81a1-4612-b1d4-2112c4e3f510 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:41.508380 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca69d51-81a1-4612-b1d4-2112c4e3f510 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:41.509953 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 516/1046] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:40:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:40:41.515694 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0b9a2e-f1c2-4fc5-9153-8cb9941e72a1 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:41.516001 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0b9a2e-f1c2-4fc5-9153-8cb9941e72a1 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:41.516251 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0b9a2e-f1c2-4fc5-9153-8cb9941e72a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:41.535875 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-be0b9a2e-f1c2-4fc5-9153-8cb9941e72a1 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:40:41.537688 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-be0b9a2e-f1c2-4fc5-9153-8cb9941e72a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:40:41.606576 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:41.614832 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 531/1047] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:40:41 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:40:41.891289 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca69d51-81a1-4612-b1d4-2112c4e3f510 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:41.995758 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca69d51-81a1-4612-b1d4-2112c4e3f510 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:42.022579 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:42.031224 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a42464e-fe54-47a0-85b2-9fc8a614da3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:42.031849 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a42464e-fe54-47a0-85b2-9fc8a614da3c None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:42.032124 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a42464e-fe54-47a0-85b2-9fc8a614da3c None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:42.032391 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a42464e-fe54-47a0-85b2-9fc8a614da3c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:42.043511 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a42464e-fe54-47a0-85b2-9fc8a614da3c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:40:42.048327 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-0a42464e-fe54-47a0-85b2-9fc8a614da3c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:40:42.048327 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:42.048327 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:42.048327 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:42.048327 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:42.048327 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:42.048327 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:42.048327 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:42.048327 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:42.048327 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:42.048327 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:42.048327 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:42.048327 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:40:42.048327 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:40:42.048327 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:40:42.048327 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:40:42.049257 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:40:42.049257 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:40:42.049257 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:40:42.049257 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:40:42.049257 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:42.049257 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:42.049257 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:42.049257 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:42.049257 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:42.049257 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:42.049257 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:42.049257 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:42.049257 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:42.049257 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:42.049257 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:42.049257 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:42.050118 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:42.050118 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:42.050118 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:42.050118 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:42.050118 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:40:42.050118 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:40:42.050118 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:40:42.050118 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:40:42.050118 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:40:42.050118 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:40:42.050118 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:40:42.050118 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:42.052904 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 517/1048] 10.222.0.27 () {56 vars in 1096 bytes} [Sun Sep 11 00:40:41 2022] GET /identity/v3/roles/admin => generated 83 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:42.055483 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be0b9a2e-f1c2-4fc5-9153-8cb9941e72a1 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:42.160546 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be0b9a2e-f1c2-4fc5-9153-8cb9941e72a1 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:42.196839 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ada626ba-4c82-40de-bc87-8e089c5aae35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:42.197444 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada626ba-4c82-40de-bc87-8e089c5aae35 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:42.197719 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada626ba-4c82-40de-bc87-8e089c5aae35 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:42.197985 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada626ba-4c82-40de-bc87-8e089c5aae35 None admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:42.198640 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ada626ba-4c82-40de-bc87-8e089c5aae35 None admin] RBAC: Adding query filter params (name=admin) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:42.198967 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ada626ba-4c82-40de-bc87-8e089c5aae35 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:42.200328 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ada626ba-4c82-40de-bc87-8e089c5aae35 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:42.208889 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 532/1049] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:40:42 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:42.214666 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a42464e-fe54-47a0-85b2-9fc8a614da3c None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:42.326321 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a42464e-fe54-47a0-85b2-9fc8a614da3c None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:42.362225 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9967b45a-809d-4898-8a98-e8a6a5432f10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:42.362889 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9967b45a-809d-4898-8a98-e8a6a5432f10 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:42.363171 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9967b45a-809d-4898-8a98-e8a6a5432f10 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:42.363449 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9967b45a-809d-4898-8a98-e8a6a5432f10 None admin] PATH_INFO: `/v3/users/69253b38ac154e7aad0de65ff9c30949` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:42.377888 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9967b45a-809d-4898-8a98-e8a6a5432f10 None admin] RBAC: Authorizing `identity:get_user(user_id=69253b38ac154e7aad0de65ff9c30949)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:42.379365 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9967b45a-809d-4898-8a98-e8a6a5432f10 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:42.381512 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 518/1050] 10.222.0.27 () {56 vars in 1177 bytes} [Sun Sep 11 00:40:42 2022] GET /identity/v3/users/69253b38ac154e7aad0de65ff9c30949 => generated 285 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:40:42.387191 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ada626ba-4c82-40de-bc87-8e089c5aae35 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:42.451074 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:42.494269 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ada626ba-4c82-40de-bc87-8e089c5aae35 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:42.527719 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c9a0ae0-8253-47fb-9b25-683615a75f13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:42.528323 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9a0ae0-8253-47fb-9b25-683615a75f13 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:42.528602 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9a0ae0-8253-47fb-9b25-683615a75f13 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:42.528875 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9a0ae0-8253-47fb-9b25-683615a75f13 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:42.595364 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c9a0ae0-8253-47fb-9b25-683615a75f13 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:42.596804 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c9a0ae0-8253-47fb-9b25-683615a75f13 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:42.657474 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:42.709658 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 533/1051] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:40:42 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 11 00:40:42.710962 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9967b45a-809d-4898-8a98-e8a6a5432f10 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:42.758172 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:42.766916 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa711d8f-93de-4137-b5df-bc8d50f89397 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:42.767232 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa711d8f-93de-4137-b5df-bc8d50f89397 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:42.767482 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa711d8f-93de-4137-b5df-bc8d50f89397 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:42.768974 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 534/1052] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:40:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 11 00:40:42.774731 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc344e00-98f1-4890-9971-e00cb72e8d4a None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:42.775096 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc344e00-98f1-4890-9971-e00cb72e8d4a None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:42.775314 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc344e00-98f1-4890-9971-e00cb72e8d4a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:42.793442 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-dc344e00-98f1-4890-9971-e00cb72e8d4a None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:40:42.794621 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-dc344e00-98f1-4890-9971-e00cb72e8d4a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:40:42.814258 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9967b45a-809d-4898-8a98-e8a6a5432f10 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:42.853369 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd07dffd-29a8-4dd0-92cc-57d90cb45049 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:42.853995 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd07dffd-29a8-4dd0-92cc-57d90cb45049 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:42.854274 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd07dffd-29a8-4dd0-92cc-57d90cb45049 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:42.854543 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd07dffd-29a8-4dd0-92cc-57d90cb45049 None admin] PATH_INFO: `/v3/projects/b7316adb92cd4d9ba42313ea22a74d3d` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:42.855683 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd07dffd-29a8-4dd0-92cc-57d90cb45049 None admin] RBAC: Authorizing `identity:get_project(project_id=b7316adb92cd4d9ba42313ea22a74d3d)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:42.857110 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd07dffd-29a8-4dd0-92cc-57d90cb45049 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:42.863745 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 519/1053] 10.222.0.27 () {56 vars in 1186 bytes} [Sun Sep 11 00:40:42 2022] GET /identity/v3/projects/b7316adb92cd4d9ba42313ea22a74d3d => generated 311 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:42.864929 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd07dffd-29a8-4dd0-92cc-57d90cb45049 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:42.870238 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 535/1054] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:40:42 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 11 00:40:42.965856 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd07dffd-29a8-4dd0-92cc-57d90cb45049 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:42.979473 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:43.001175 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-642d4b4d-f6ab-4759-9096-f973f071a1d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:43.001773 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642d4b4d-f6ab-4759-9096-f973f071a1d7 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:43.002048 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642d4b4d-f6ab-4759-9096-f973f071a1d7 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:43.002317 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642d4b4d-f6ab-4759-9096-f973f071a1d7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:43.003063 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-642d4b4d-f6ab-4759-9096-f973f071a1d7 None admin] RBAC: Adding query filter params (role.id=7d9213c1187c4b5182bf8132cf23ea19, scope.project.id=b7316adb92cd4d9ba42313ea22a74d3d, user.id=69253b38ac154e7aad0de65ff9c30949) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:43.003370 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-642d4b4d-f6ab-4759-9096-f973f071a1d7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:43.005342 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-642d4b4d-f6ab-4759-9096-f973f071a1d7 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:43.010861 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 520/1055] 10.222.0.27 () {56 vars in 1374 bytes} [Sun Sep 11 00:40:42 2022] GET /identity/v3/role_assignments?user.id=69253b38ac154e7aad0de65ff9c30949&scope.project.id=b7316adb92cd4d9ba42313ea22a74d3d&role.id=7d9213c1187c4b5182bf8132cf23ea19 => generated 259 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:40:43.146383 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc344e00-98f1-4890-9971-e00cb72e8d4a None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:43.241500 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:43.251963 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc344e00-98f1-4890-9971-e00cb72e8d4a None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:43.287027 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e272d0f-8ed2-4b94-8a27-b0be010dd6ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:43.287663 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e272d0f-8ed2-4b94-8a27-b0be010dd6ea None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:43.287940 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e272d0f-8ed2-4b94-8a27-b0be010dd6ea None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:43.288208 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e272d0f-8ed2-4b94-8a27-b0be010dd6ea None admin] PATH_INFO: `/v3/services/image` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:43.288791 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e272d0f-8ed2-4b94-8a27-b0be010dd6ea None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:43.290155 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e272d0f-8ed2-4b94-8a27-b0be010dd6ea None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:43.299056 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-8e272d0f-8ed2-4b94-8a27-b0be010dd6ea None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 11 00:40:43.299056 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:43.299056 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 11 00:40:43.299056 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 11 00:40:43.299056 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 11 00:40:43.299056 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 11 00:40:43.299056 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 11 00:40:43.299056 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 11 00:40:43.299056 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 11 00:40:43.299056 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:43.299056 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 11 00:40:43.299056 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:43.299056 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:43.299056 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:43.299056 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:43.299056 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:43.299056 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:43.300063 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:43.300063 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:43.300063 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:43.300063 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:43.300063 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:43.300063 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:43.300063 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 11 00:40:43.300063 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 11 00:40:43.300063 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 11 00:40:43.300063 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 11 00:40:43.300063 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:43.300063 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:43.300063 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:43.300063 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:43.300063 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:43.300063 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:43.300946 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:43.300946 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:43.300946 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:43.300946 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:43.300946 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:43.300946 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:43.300946 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:43.300946 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:43.300946 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:43.300946 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:43.300946 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 11 00:40:43.300946 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 11 00:40:43.300946 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 11 00:40:43.300946 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:43.300946 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 536/1056] 10.222.0.27 () {56 vars in 1105 bytes} [Sun Sep 11 00:40:43 2022] GET /identity/v3/services/image => generated 86 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:40:43.308631 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-642d4b4d-f6ab-4759-9096-f973f071a1d7 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:43.410674 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-642d4b4d-f6ab-4759-9096-f973f071a1d7 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:43.445744 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1577ff7c-3269-4138-bdb8-d0202dfc475f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:43.446351 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1577ff7c-3269-4138-bdb8-d0202dfc475f None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:43.446627 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1577ff7c-3269-4138-bdb8-d0202dfc475f None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:43.446926 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1577ff7c-3269-4138-bdb8-d0202dfc475f None admin] PATH_INFO: `/v3/services` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:43.447551 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1577ff7c-3269-4138-bdb8-d0202dfc475f None admin] RBAC: Adding query filter params (name=image) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:43.447854 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1577ff7c-3269-4138-bdb8-d0202dfc475f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:43.449222 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1577ff7c-3269-4138-bdb8-d0202dfc475f None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:43.461068 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 521/1057] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:40:43 2022] GET /identity/v3/services?name=image => generated 122 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:40:43.461269 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e272d0f-8ed2-4b94-8a27-b0be010dd6ea None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:43.570903 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e272d0f-8ed2-4b94-8a27-b0be010dd6ea None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:43.604924 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c30712-88ac-4c59-b5c2-1392c468132e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:43.605524 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c30712-88ac-4c59-b5c2-1392c468132e None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:43.605797 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c30712-88ac-4c59-b5c2-1392c468132e None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:43.606063 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c30712-88ac-4c59-b5c2-1392c468132e None admin] PATH_INFO: `/v3/services` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:43.606820 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c30712-88ac-4c59-b5c2-1392c468132e None admin] RBAC: Adding query filter params (type=image) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:43.607121 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c30712-88ac-4c59-b5c2-1392c468132e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:43.608475 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c30712-88ac-4c59-b5c2-1392c468132e None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:43.618091 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 537/1058] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:40:43 2022] GET /identity/v3/services?type=image => generated 350 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:40:43.619078 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1577ff7c-3269-4138-bdb8-d0202dfc475f None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:43.642756 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2445e06e-beb9-4922-8aa0-df61bc7a161d None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:43.643114 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2445e06e-beb9-4922-8aa0-df61bc7a161d None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:43.643349 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2445e06e-beb9-4922-8aa0-df61bc7a161d None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:43.644826 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 538/1059] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:40:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 11 00:40:43.650270 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14de3595-3cd6-4b40-948a-7d6f7aa11b04 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:43.650556 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14de3595-3cd6-4b40-948a-7d6f7aa11b04 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:43.650854 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14de3595-3cd6-4b40-948a-7d6f7aa11b04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:43.667786 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:43.670561 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-14de3595-3cd6-4b40-948a-7d6f7aa11b04 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:40:43.671800 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-14de3595-3cd6-4b40-948a-7d6f7aa11b04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:40:43.686686 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:43.735800 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1577ff7c-3269-4138-bdb8-d0202dfc475f None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:43.746839 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 539/1060] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:40:43 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 11 00:40:43.774965 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e331cf53-ffc0-499a-b662-2bdde279084d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:43.775571 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e331cf53-ffc0-499a-b662-2bdde279084d None admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:43.775850 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e331cf53-ffc0-499a-b662-2bdde279084d None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:43.776123 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e331cf53-ffc0-499a-b662-2bdde279084d None admin] PATH_INFO: `/v3/endpoints` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:43.776787 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e331cf53-ffc0-499a-b662-2bdde279084d None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:43.778185 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e331cf53-ffc0-499a-b662-2bdde279084d None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:43.799578 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 522/1061] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:40:43 2022] POST /identity/v3/endpoints => generated 327 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:40:44.016619 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14de3595-3cd6-4b40-948a-7d6f7aa11b04 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:44.125972 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14de3595-3cd6-4b40-948a-7d6f7aa11b04 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:44.166342 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5786164-d7f1-4807-86aa-b75acfd05685 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:44.166944 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5786164-d7f1-4807-86aa-b75acfd05685 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:44.167226 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5786164-d7f1-4807-86aa-b75acfd05685 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:44.167496 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5786164-d7f1-4807-86aa-b75acfd05685 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:44.178425 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5786164-d7f1-4807-86aa-b75acfd05685 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:40:44.183293 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-e5786164-d7f1-4807-86aa-b75acfd05685 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 11 00:40:44.183293 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:44.183293 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:44.183293 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:44.183293 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:44.183293 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:44.183293 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:44.183293 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:44.183293 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:44.183293 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:44.183293 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:44.183293 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:44.183293 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:40:44.183293 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:40:44.183293 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:40:44.183293 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:40:44.184245 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:40:44.184245 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:40:44.184245 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:40:44.184245 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:40:44.184245 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:44.184245 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:44.184245 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:44.184245 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:44.184245 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:44.184245 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:44.184245 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:44.184245 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:44.184245 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:44.184245 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:44.184245 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:44.184245 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:44.185145 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:44.185145 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:44.185145 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:44.185145 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:44.185145 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:40:44.185145 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:40:44.185145 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:40:44.185145 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:40:44.185145 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:40:44.185145 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:40:44.185145 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 11 00:40:44.185145 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:44.186425 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 540/1062] 10.222.0.27 () {56 vars in 1120 bytes} [Sun Sep 11 00:40:44 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:44.190526 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e331cf53-ffc0-499a-b662-2bdde279084d None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:44.294552 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:44.300263 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e331cf53-ffc0-499a-b662-2bdde279084d None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:44.336029 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52763f1d-0ccc-4495-943a-e8d56e4b06b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:44.336637 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52763f1d-0ccc-4495-943a-e8d56e4b06b7 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:44.336911 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52763f1d-0ccc-4495-943a-e8d56e4b06b7 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:44.337176 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52763f1d-0ccc-4495-943a-e8d56e4b06b7 None admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:44.337834 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52763f1d-0ccc-4495-943a-e8d56e4b06b7 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:44.338133 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52763f1d-0ccc-4495-943a-e8d56e4b06b7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:44.339516 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52763f1d-0ccc-4495-943a-e8d56e4b06b7 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:44.348253 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 523/1063] 10.222.0.27 () {56 vars in 1115 bytes} [Sun Sep 11 00:40:44 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:44.353696 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5786164-d7f1-4807-86aa-b75acfd05685 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:44.358972 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:44.456547 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5786164-d7f1-4807-86aa-b75acfd05685 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:44.491612 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6732918a-6f45-4885-b518-e54a339417dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:44.492267 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6732918a-6f45-4885-b518-e54a339417dd None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:44.492546 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6732918a-6f45-4885-b518-e54a339417dd None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:44.492815 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6732918a-6f45-4885-b518-e54a339417dd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:44.502817 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6732918a-6f45-4885-b518-e54a339417dd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:44.504280 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6732918a-6f45-4885-b518-e54a339417dd None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:44.519264 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-6732918a-6f45-4885-b518-e54a339417dd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:40:44.519264 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:44.519264 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:44.519264 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:44.519264 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:44.519264 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:44.519264 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:44.519264 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:44.519264 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:44.519264 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:44.519264 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:44.519264 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:44.519264 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:40:44.519264 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:40:44.519264 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:40:44.519264 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:40:44.520446 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:44.520446 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:44.520446 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:44.520446 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:44.520446 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:44.520446 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:44.520446 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:44.520446 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:44.520446 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:44.520446 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:44.520446 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:44.520446 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:44.520446 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:44.520446 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:44.520446 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:44.520446 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:44.520446 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:40:44.521454 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:40:44.521454 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:40:44.521454 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:44.523120 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 541/1064] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:40:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:44.526191 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52763f1d-0ccc-4495-943a-e8d56e4b06b7 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:44.626966 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52763f1d-0ccc-4495-943a-e8d56e4b06b7 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:44.661718 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939335a0-477d-4ba7-959b-55415803198c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:44.662318 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939335a0-477d-4ba7-959b-55415803198c None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:44.662593 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939335a0-477d-4ba7-959b-55415803198c None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:44.662890 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939335a0-477d-4ba7-959b-55415803198c None admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:44.663531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-939335a0-477d-4ba7-959b-55415803198c None admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:44.663831 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-939335a0-477d-4ba7-959b-55415803198c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:44.665201 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-939335a0-477d-4ba7-959b-55415803198c None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:44.681358 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 524/1065] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:40:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:44.682961 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6732918a-6f45-4885-b518-e54a339417dd None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:44.745557 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:44.791949 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6732918a-6f45-4885-b518-e54a339417dd None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:44.825377 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-202e8c3e-c30a-495c-af60-38381931a7c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:44.826008 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202e8c3e-c30a-495c-af60-38381931a7c3 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:44.826289 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202e8c3e-c30a-495c-af60-38381931a7c3 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:44.826559 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202e8c3e-c30a-495c-af60-38381931a7c3 None admin] PATH_INFO: `/v3/users/nova` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:44.835177 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-202e8c3e-c30a-495c-af60-38381931a7c3 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:44.836620 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-202e8c3e-c30a-495c-af60-38381931a7c3 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:44.847974 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-202e8c3e-c30a-495c-af60-38381931a7c3 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 11 00:40:44.847974 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:44.847974 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:44.847974 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:44.847974 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:44.847974 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:44.847974 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:44.847974 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:44.847974 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:44.847974 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:44.847974 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:44.847974 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:44.847974 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 11 00:40:44.847974 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 11 00:40:44.847974 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 11 00:40:44.847974 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 11 00:40:44.849230 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:44.849230 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:44.849230 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 11 00:40:44.849230 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:40:44.849230 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 11 00:40:44.849230 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:40:44.849230 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:44.849230 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:44.849230 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:44.849230 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:44.849230 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:44.849230 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:44.849230 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:44.849230 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:44.849230 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:44.849230 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:44.849230 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:44.849230 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:44.850211 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:44.850211 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:44.850211 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 11 00:40:44.850211 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 11 00:40:44.850211 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 11 00:40:44.850211 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 11 00:40:44.850211 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 11 00:40:44.850211 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 11 00:40:44.850211 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 11 00:40:44.850211 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:44.853983 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 542/1066] 10.222.0.27 () {56 vars in 1093 bytes} [Sun Sep 11 00:40:44 2022] GET /identity/v3/users/nova => generated 82 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:44.855437 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939335a0-477d-4ba7-959b-55415803198c None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:44.878466 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:44.960304 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939335a0-477d-4ba7-959b-55415803198c None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:44.993931 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e70b00-1d76-4aaf-aaeb-b8222e989253 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:44.994535 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e70b00-1d76-4aaf-aaeb-b8222e989253 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:44.994832 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e70b00-1d76-4aaf-aaeb-b8222e989253 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:44.995107 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e70b00-1d76-4aaf-aaeb-b8222e989253 None admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:44.995835 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e70b00-1d76-4aaf-aaeb-b8222e989253 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:44.996135 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e70b00-1d76-4aaf-aaeb-b8222e989253 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:44.997560 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e70b00-1d76-4aaf-aaeb-b8222e989253 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:45.020733 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 525/1067] 10.222.0.27 () {56 vars in 1133 bytes} [Sun Sep 11 00:40:44 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:45.022485 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-202e8c3e-c30a-495c-af60-38381931a7c3 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:45.140892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-202e8c3e-c30a-495c-af60-38381931a7c3 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:45.157300 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:45.177108 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc57f95-9403-402b-8d15-324c0d4074c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:45.177750 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc57f95-9403-402b-8d15-324c0d4074c0 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:45.178028 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc57f95-9403-402b-8d15-324c0d4074c0 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:45.178300 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc57f95-9403-402b-8d15-324c0d4074c0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:45.182710 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-132039e2-c604-4710-b501-af1066e32f68 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:45.183028 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-132039e2-c604-4710-b501-af1066e32f68 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:45.183303 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-132039e2-c604-4710-b501-af1066e32f68 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:45.184773 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 526/1068] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:40:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 11 00:40:45.189511 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5e2032-b41c-4bef-bf35-b325a0448d86 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:45.189797 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5e2032-b41c-4bef-bf35-b325a0448d86 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:45.190081 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5e2032-b41c-4bef-bf35-b325a0448d86 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:45.190915 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cc57f95-9403-402b-8d15-324c0d4074c0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:45.193115 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cc57f95-9403-402b-8d15-324c0d4074c0 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:45.207695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-4cc57f95-9403-402b-8d15-324c0d4074c0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:40:45.207695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:45.207695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:45.207695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:45.207695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:45.207695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:45.207695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:45.207695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:45.207695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:45.207695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:45.207695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:45.207695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:45.207695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:40:45.207695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:40:45.207695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:40:45.207695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:40:45.209051 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:45.209051 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:45.209051 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:45.209051 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:45.209051 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:45.209051 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:45.209051 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:45.209051 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:45.209051 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:45.209051 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:45.209051 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:45.209051 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:45.209051 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:45.209051 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:45.209051 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:45.209051 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:45.209051 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:40:45.210278 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:40:45.210278 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:40:45.210278 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:45.210486 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-2c5e2032-b41c-4bef-bf35-b325a0448d86 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:40:45.210486 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-2c5e2032-b41c-4bef-bf35-b325a0448d86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:40:45.216691 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 543/1069] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:40:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:45.218055 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc57f95-9403-402b-8d15-324c0d4074c0 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:45.282941 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 527/1070] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:40:45 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 11 00:40:45.319402 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc57f95-9403-402b-8d15-324c0d4074c0 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:45.351795 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de536f2-78e4-4bc3-8e89-858fea74cef6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:45.352389 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de536f2-78e4-4bc3-8e89-858fea74cef6 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:45.352659 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de536f2-78e4-4bc3-8e89-858fea74cef6 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:45.352923 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de536f2-78e4-4bc3-8e89-858fea74cef6 None admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:45.353550 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9de536f2-78e4-4bc3-8e89-858fea74cef6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:45.353845 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9de536f2-78e4-4bc3-8e89-858fea74cef6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:45.355220 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9de536f2-78e4-4bc3-8e89-858fea74cef6 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:45.371517 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 544/1071] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:40:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:45.372957 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c5e2032-b41c-4bef-bf35-b325a0448d86 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:45.384237 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:45.478008 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c5e2032-b41c-4bef-bf35-b325a0448d86 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:45.513573 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bccbedc-befb-4f3b-88cc-aeed8f647ca4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:45.514195 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bccbedc-befb-4f3b-88cc-aeed8f647ca4 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:45.514467 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bccbedc-befb-4f3b-88cc-aeed8f647ca4 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:45.514731 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bccbedc-befb-4f3b-88cc-aeed8f647ca4 None admin] PATH_INFO: `/v3/projects/service` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:45.521717 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bccbedc-befb-4f3b-88cc-aeed8f647ca4 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:45.523193 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bccbedc-befb-4f3b-88cc-aeed8f647ca4 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:45.535647 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-2bccbedc-befb-4f3b-88cc-aeed8f647ca4 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:40:45.535647 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:45.535647 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:45.535647 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:45.535647 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:45.535647 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:45.535647 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:45.535647 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:45.535647 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:45.535647 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:45.535647 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:45.535647 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:45.535647 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 11 00:40:45.535647 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 11 00:40:45.535647 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 11 00:40:45.536876 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 11 00:40:45.536876 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:45.536876 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:45.536876 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:45.536876 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:45.536876 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:45.536876 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:45.536876 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:45.536876 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:45.536876 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:45.536876 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:45.536876 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:45.536876 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:45.536876 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:45.536876 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:45.536876 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:45.536876 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:45.537800 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 11 00:40:45.537800 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 11 00:40:45.537800 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 11 00:40:45.537800 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 11 00:40:45.537800 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 11 00:40:45.537800 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 11 00:40:45.537800 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:40:45.537800 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:45.537800 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 528/1072] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:40:45 2022] GET /identity/v3/projects/service => generated 88 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:45.544112 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de536f2-78e4-4bc3-8e89-858fea74cef6 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:45.549511 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bccbedc-befb-4f3b-88cc-aeed8f647ca4 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:45.573920 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:45.622001 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:45.651200 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de536f2-78e4-4bc3-8e89-858fea74cef6 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:45.660061 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bccbedc-befb-4f3b-88cc-aeed8f647ca4 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:45.685010 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a651129-7292-4aa1-9a45-dc4532ccfacc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:45.685612 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a651129-7292-4aa1-9a45-dc4532ccfacc None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:45.685881 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a651129-7292-4aa1-9a45-dc4532ccfacc None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:45.686149 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a651129-7292-4aa1-9a45-dc4532ccfacc None admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:45.686861 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a651129-7292-4aa1-9a45-dc4532ccfacc None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:45.687156 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a651129-7292-4aa1-9a45-dc4532ccfacc None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:45.688595 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a651129-7292-4aa1-9a45-dc4532ccfacc None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:45.698047 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba1e8327-471a-4a43-8f4c-8c4cfb594e26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:45.698968 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1e8327-471a-4a43-8f4c-8c4cfb594e26 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:45.699427 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1e8327-471a-4a43-8f4c-8c4cfb594e26 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:45.699893 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1e8327-471a-4a43-8f4c-8c4cfb594e26 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:45.700490 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 545/1073] 10.222.0.27 () {56 vars in 1148 bytes} [Sun Sep 11 00:40:45 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:45.706282 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a651129-7292-4aa1-9a45-dc4532ccfacc None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:45.716410 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba1e8327-471a-4a43-8f4c-8c4cfb594e26 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:40:45.721676 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-ba1e8327-471a-4a43-8f4c-8c4cfb594e26 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:40:45.721676 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:45.721676 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:45.721676 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:45.721676 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:45.721676 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:45.721676 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:45.721676 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:45.721676 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:45.721676 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:45.721676 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:45.721676 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:45.721676 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:40:45.721676 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:40:45.721676 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:40:45.721676 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:40:45.722633 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:40:45.722633 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:40:45.722633 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:40:45.722633 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:40:45.722633 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:45.722633 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:45.722633 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:45.722633 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:45.722633 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:45.722633 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:45.722633 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:45.722633 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:45.722633 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:45.722633 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:45.722633 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:45.722633 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:45.723628 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:45.723628 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:45.723628 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:45.723628 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:45.723628 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:40:45.723628 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:40:45.723628 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:40:45.723628 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:40:45.723628 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:40:45.723628 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:40:45.723628 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:40:45.723628 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:45.723628 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 529/1074] 10.222.0.27 () {56 vars in 1096 bytes} [Sun Sep 11 00:40:45 2022] GET /identity/v3/roles/admin => generated 83 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:40:45.729495 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba1e8327-471a-4a43-8f4c-8c4cfb594e26 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:45.800329 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:45.813196 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a651129-7292-4aa1-9a45-dc4532ccfacc None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:45.840116 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba1e8327-471a-4a43-8f4c-8c4cfb594e26 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:45.847262 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad98c750-00e3-4ed6-bb1f-a23727ca6d4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:45.847961 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad98c750-00e3-4ed6-bb1f-a23727ca6d4a None admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:45.848746 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad98c750-00e3-4ed6-bb1f-a23727ca6d4a None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:45.849011 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad98c750-00e3-4ed6-bb1f-a23727ca6d4a None admin] PATH_INFO: `/v3/projects/b504a5058bc6474c8cc077dfeec16799/users/835def0e6c6d4365a2767051509eb0d6/roles/a6b5856c8f5c4144b39cab1b2916ac93` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:45.851375 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad98c750-00e3-4ed6-bb1f-a23727ca6d4a None admin] RBAC: Authorizing `identity:create_grant(project_id=b504a5058bc6474c8cc077dfeec16799, user_id=835def0e6c6d4365a2767051509eb0d6, role_id=a6b5856c8f5c4144b39cab1b2916ac93)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:45.852820 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad98c750-00e3-4ed6-bb1f-a23727ca6d4a None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:45.867064 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 546/1075] 10.222.0.27 () {58 vars in 1439 bytes} [Sun Sep 11 00:40:45 2022] PUT /identity/v3/projects/b504a5058bc6474c8cc077dfeec16799/users/835def0e6c6d4365a2767051509eb0d6/roles/a6b5856c8f5c4144b39cab1b2916ac93 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:40:45.882339 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:45.891495 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a962b84-b09b-4203-9b15-f27ed3fcbd20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:45.892124 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a962b84-b09b-4203-9b15-f27ed3fcbd20 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:45.892397 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a962b84-b09b-4203-9b15-f27ed3fcbd20 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:45.892664 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a962b84-b09b-4203-9b15-f27ed3fcbd20 None admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:45.893346 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a962b84-b09b-4203-9b15-f27ed3fcbd20 None admin] RBAC: Adding query filter params (name=admin) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:45.893647 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a962b84-b09b-4203-9b15-f27ed3fcbd20 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:45.895295 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a962b84-b09b-4203-9b15-f27ed3fcbd20 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:45.910456 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 530/1076] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:40:45 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:40:45.912340 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad98c750-00e3-4ed6-bb1f-a23727ca6d4a None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:46.027554 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad98c750-00e3-4ed6-bb1f-a23727ca6d4a None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:46.049635 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af87b40-2b2e-4b44-bf4e-d2eb7b98e565 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:46.049923 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af87b40-2b2e-4b44-bf4e-d2eb7b98e565 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:46.050189 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af87b40-2b2e-4b44-bf4e-d2eb7b98e565 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:46.052167 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 531/1077] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:40:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:40:46.058619 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b96a01-0b0c-4a3c-b846-8162bd82c47f None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:46.058944 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b96a01-0b0c-4a3c-b846-8162bd82c47f None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:46.059222 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b96a01-0b0c-4a3c-b846-8162bd82c47f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:46.073131 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a970d39-7f06-47d1-9a0d-1c17e02c0fc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:46.073769 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a970d39-7f06-47d1-9a0d-1c17e02c0fc5 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:46.074050 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a970d39-7f06-47d1-9a0d-1c17e02c0fc5 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:46.074318 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a970d39-7f06-47d1-9a0d-1c17e02c0fc5 None admin] PATH_INFO: `/v3/users/69253b38ac154e7aad0de65ff9c30949` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:46.075552 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a970d39-7f06-47d1-9a0d-1c17e02c0fc5 None admin] RBAC: Authorizing `identity:get_user(user_id=69253b38ac154e7aad0de65ff9c30949)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:46.077651 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a970d39-7f06-47d1-9a0d-1c17e02c0fc5 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:46.079754 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 547/1078] 10.222.0.27 () {56 vars in 1177 bytes} [Sun Sep 11 00:40:45 2022] GET /identity/v3/users/69253b38ac154e7aad0de65ff9c30949 => generated 285 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:46.082304 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-70b96a01-0b0c-4a3c-b846-8162bd82c47f None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:40:46.084281 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-70b96a01-0b0c-4a3c-b846-8162bd82c47f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:40:46.085848 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a970d39-7f06-47d1-9a0d-1c17e02c0fc5 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:46.107324 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:46.162297 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 532/1079] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:40:46 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:40:46.195297 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a970d39-7f06-47d1-9a0d-1c17e02c0fc5 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:46.230417 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a0d0c4-ee43-49f2-bf03-fd298ce6f5a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:46.231082 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a0d0c4-ee43-49f2-bf03-fd298ce6f5a0 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:46.231366 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a0d0c4-ee43-49f2-bf03-fd298ce6f5a0 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:46.231640 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a0d0c4-ee43-49f2-bf03-fd298ce6f5a0 None admin] PATH_INFO: `/v3/projects/b7316adb92cd4d9ba42313ea22a74d3d` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:46.232756 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0a0d0c4-ee43-49f2-bf03-fd298ce6f5a0 None admin] RBAC: Authorizing `identity:get_project(project_id=b7316adb92cd4d9ba42313ea22a74d3d)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:46.234188 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0a0d0c4-ee43-49f2-bf03-fd298ce6f5a0 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:46.236057 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 548/1080] 10.222.0.27 () {56 vars in 1186 bytes} [Sun Sep 11 00:40:46 2022] GET /identity/v3/projects/b7316adb92cd4d9ba42313ea22a74d3d => generated 311 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:40:46.242437 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70b96a01-0b0c-4a3c-b846-8162bd82c47f None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:46.342761 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70b96a01-0b0c-4a3c-b846-8162bd82c47f None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:46.366852 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:46.382100 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2756f8f7-e12f-447b-bc33-d1c4aaec00a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:46.382760 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2756f8f7-e12f-447b-bc33-d1c4aaec00a8 None admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:46.383066 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2756f8f7-e12f-447b-bc33-d1c4aaec00a8 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:46.383335 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2756f8f7-e12f-447b-bc33-d1c4aaec00a8 None admin] PATH_INFO: `/v3/projects/b7316adb92cd4d9ba42313ea22a74d3d/users/69253b38ac154e7aad0de65ff9c30949/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:46.385212 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2756f8f7-e12f-447b-bc33-d1c4aaec00a8 None admin] RBAC: Authorizing `identity:create_grant(project_id=b7316adb92cd4d9ba42313ea22a74d3d, user_id=69253b38ac154e7aad0de65ff9c30949, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:46.386647 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2756f8f7-e12f-447b-bc33-d1c4aaec00a8 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:46.399556 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 533/1081] 10.222.0.27 () {58 vars in 1439 bytes} [Sun Sep 11 00:40:46 2022] PUT /identity/v3/projects/b7316adb92cd4d9ba42313ea22a74d3d/users/69253b38ac154e7aad0de65ff9c30949/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:40:46.443585 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a0d0c4-ee43-49f2-bf03-fd298ce6f5a0 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:46.541326 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:46.560120 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a0d0c4-ee43-49f2-bf03-fd298ce6f5a0 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:46.601680 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41fc668f-a96f-4ef9-a6bd-380db7c70c7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:46.602215 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41fc668f-a96f-4ef9-a6bd-380db7c70c7b None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:46.602491 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41fc668f-a96f-4ef9-a6bd-380db7c70c7b None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:46.602758 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41fc668f-a96f-4ef9-a6bd-380db7c70c7b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:46.671602 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41fc668f-a96f-4ef9-a6bd-380db7c70c7b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:46.673056 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41fc668f-a96f-4ef9-a6bd-380db7c70c7b None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:46.755942 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:46.775974 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 549/1082] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:40:46 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 333 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 11 00:40:46.781544 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2756f8f7-e12f-447b-bc33-d1c4aaec00a8 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:46.883751 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2756f8f7-e12f-447b-bc33-d1c4aaec00a8 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:46.918777 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a796be-cb4c-410b-9e42-44456088390e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:46.919435 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a796be-cb4c-410b-9e42-44456088390e None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:46.919710 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a796be-cb4c-410b-9e42-44456088390e None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:46.919976 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a796be-cb4c-410b-9e42-44456088390e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:46.930229 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22a796be-cb4c-410b-9e42-44456088390e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:46.931708 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22a796be-cb4c-410b-9e42-44456088390e None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:46.946914 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-22a796be-cb4c-410b-9e42-44456088390e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:40:46.946914 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:46.946914 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:46.946914 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:46.946914 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:46.946914 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:46.946914 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:46.946914 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:46.946914 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:46.946914 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:46.946914 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:46.946914 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:46.946914 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:40:46.946914 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:40:46.946914 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:40:46.946914 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:40:46.947931 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:46.947931 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:46.947931 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:46.947931 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:46.947931 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:46.947931 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:46.947931 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:46.947931 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:46.947931 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:46.947931 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:46.947931 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:46.947931 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:46.947931 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:46.947931 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:46.947931 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:46.947931 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:46.947931 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:40:46.948868 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:40:46.948868 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:40:46.948868 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:46.948868 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 534/1083] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:40:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:46.953883 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41fc668f-a96f-4ef9-a6bd-380db7c70c7b None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:47.056631 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41fc668f-a96f-4ef9-a6bd-380db7c70c7b None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:47.092501 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d543fca-32a7-4da9-ba41-2093cbdfdc86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:47.093117 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d543fca-32a7-4da9-ba41-2093cbdfdc86 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:47.093398 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d543fca-32a7-4da9-ba41-2093cbdfdc86 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:47.093670 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d543fca-32a7-4da9-ba41-2093cbdfdc86 None admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:47.094318 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d543fca-32a7-4da9-ba41-2093cbdfdc86 None admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:47.094622 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d543fca-32a7-4da9-ba41-2093cbdfdc86 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:47.096022 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d543fca-32a7-4da9-ba41-2093cbdfdc86 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:47.110613 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 550/1084] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:40:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:48.045763 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b6b609-b6ac-471a-9929-367c36eab967 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:48.046337 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b6b609-b6ac-471a-9929-367c36eab967 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:48.046337 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b6b609-b6ac-471a-9929-367c36eab967 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:48.047670 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 535/1085] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:40:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:40:48.053012 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e02d14-04ea-4d24-89ed-acafe794000d None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:48.053426 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e02d14-04ea-4d24-89ed-acafe794000d None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:48.053677 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e02d14-04ea-4d24-89ed-acafe794000d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:48.072199 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-29e02d14-04ea-4d24-89ed-acafe794000d None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:40:48.073358 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-29e02d14-04ea-4d24-89ed-acafe794000d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:40:48.114557 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:48.140722 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 551/1086] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:40:48 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:40:48.409361 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b6b609-b6ac-471a-9929-367c36eab967 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:48.440878 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:48.516613 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b6b609-b6ac-471a-9929-367c36eab967 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:48.549546 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-550b9c2f-e0fb-4b78-9702-9f582f6b5449 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:48.550160 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550b9c2f-e0fb-4b78-9702-9f582f6b5449 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:48.550432 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550b9c2f-e0fb-4b78-9702-9f582f6b5449 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:48.550699 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550b9c2f-e0fb-4b78-9702-9f582f6b5449 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:48.561650 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-550b9c2f-e0fb-4b78-9702-9f582f6b5449 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:40:48.566379 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-550b9c2f-e0fb-4b78-9702-9f582f6b5449 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 11 00:40:48.566379 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:48.566379 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:48.566379 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:48.566379 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:48.566379 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:48.566379 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:48.566379 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:48.566379 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:48.566379 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:48.566379 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:48.566379 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:48.566379 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:40:48.566379 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:40:48.566379 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:40:48.566379 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:40:48.568142 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:40:48.568142 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:40:48.568142 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:40:48.568142 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:40:48.568142 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:48.568142 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:48.568142 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:48.568142 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:48.568142 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:48.568142 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:48.568142 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:48.568142 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:48.568142 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:48.568142 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:48.568142 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:48.568142 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:48.569017 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:48.569017 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:48.569017 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:48.569017 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:48.569017 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:40:48.569017 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:40:48.569017 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:40:48.569017 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:40:48.569017 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:40:48.569017 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:40:48.569017 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 11 00:40:48.569017 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:48.569017 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 536/1087] 10.222.0.27 () {56 vars in 1120 bytes} [Sun Sep 11 00:40:48 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:40:48.573988 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e02d14-04ea-4d24-89ed-acafe794000d None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:48.592203 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ca83ae-4889-410d-b76b-ca008a6bef1a None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:48.592472 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ca83ae-4889-410d-b76b-ca008a6bef1a None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:48.592749 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ca83ae-4889-410d-b76b-ca008a6bef1a None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:48.594077 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 537/1088] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:40:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 11 00:40:48.599639 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de38481a-5ef4-4e62-b768-946dd2471cf2 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:48.599920 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de38481a-5ef4-4e62-b768-946dd2471cf2 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:48.600193 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de38481a-5ef4-4e62-b768-946dd2471cf2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:48.618751 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-de38481a-5ef4-4e62-b768-946dd2471cf2 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:40:48.619971 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-de38481a-5ef4-4e62-b768-946dd2471cf2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:40:48.671961 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e02d14-04ea-4d24-89ed-acafe794000d None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:48.685281 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 538/1089] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:40:48 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 11 00:40:48.705154 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c41e76-2dd4-415b-80c9-631d0a44ae2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:48.705754 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c41e76-2dd4-415b-80c9-631d0a44ae2e None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:48.706028 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c41e76-2dd4-415b-80c9-631d0a44ae2e None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:48.706295 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c41e76-2dd4-415b-80c9-631d0a44ae2e None admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:48.706973 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67c41e76-2dd4-415b-80c9-631d0a44ae2e None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:48.707270 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67c41e76-2dd4-415b-80c9-631d0a44ae2e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:48.709396 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67c41e76-2dd4-415b-80c9-631d0a44ae2e None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:48.717013 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 552/1090] 10.222.0.27 () {56 vars in 1115 bytes} [Sun Sep 11 00:40:48 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:48.722776 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de38481a-5ef4-4e62-b768-946dd2471cf2 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:48.746229 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:48.820105 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de38481a-5ef4-4e62-b768-946dd2471cf2 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:48.852819 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e3aa3f9-fdcd-4a9a-8945-93f95a6104a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:48.853431 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3aa3f9-fdcd-4a9a-8945-93f95a6104a0 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:48.853702 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3aa3f9-fdcd-4a9a-8945-93f95a6104a0 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:48.853968 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3aa3f9-fdcd-4a9a-8945-93f95a6104a0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:48.863616 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e3aa3f9-fdcd-4a9a-8945-93f95a6104a0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:48.865482 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e3aa3f9-fdcd-4a9a-8945-93f95a6104a0 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:48.880166 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-9e3aa3f9-fdcd-4a9a-8945-93f95a6104a0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:40:48.880166 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:48.880166 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:48.880166 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:48.880166 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:48.880166 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:48.880166 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:48.880166 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:48.880166 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:48.880166 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:48.880166 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:48.880166 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:48.880166 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:40:48.880166 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:40:48.880166 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:40:48.880166 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:40:48.885853 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:48.885853 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:48.885853 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:48.885853 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:48.885853 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:48.885853 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:48.885853 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:48.885853 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:48.885853 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:48.885853 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:48.885853 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:48.885853 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:48.885853 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:48.885853 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:48.885853 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:48.885853 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:48.885853 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:40:48.886869 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:40:48.886869 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:40:48.886869 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:48.886869 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 539/1091] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:40:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:40:48.887128 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c41e76-2dd4-415b-80c9-631d0a44ae2e None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:48.921916 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:48.956174 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e3aa3f9-fdcd-4a9a-8945-93f95a6104a0 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:48.987179 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c41e76-2dd4-415b-80c9-631d0a44ae2e None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:49.021384 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799d42a3-4513-4c03-b76f-81fcc289e033 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:49.021976 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799d42a3-4513-4c03-b76f-81fcc289e033 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:49.022249 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799d42a3-4513-4c03-b76f-81fcc289e033 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:49.022513 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799d42a3-4513-4c03-b76f-81fcc289e033 None admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:49.023170 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-799d42a3-4513-4c03-b76f-81fcc289e033 None admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:49.023467 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-799d42a3-4513-4c03-b76f-81fcc289e033 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:49.024818 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-799d42a3-4513-4c03-b76f-81fcc289e033 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:49.032291 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:49.037639 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 553/1092] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:40:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:40:49.043229 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799d42a3-4513-4c03-b76f-81fcc289e033 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:49.058915 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e3aa3f9-fdcd-4a9a-8945-93f95a6104a0 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:49.095916 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-886db399-c2b3-4d41-845a-29abdfe4793e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:49.096541 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886db399-c2b3-4d41-845a-29abdfe4793e None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:49.096817 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886db399-c2b3-4d41-845a-29abdfe4793e None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:49.097088 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886db399-c2b3-4d41-845a-29abdfe4793e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:49.110957 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-886db399-c2b3-4d41-845a-29abdfe4793e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:40:49.116932 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-886db399-c2b3-4d41-845a-29abdfe4793e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:40:49.116932 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:49.116932 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:49.116932 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:49.116932 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:49.116932 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:49.116932 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:49.116932 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:49.116932 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:49.116932 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:49.116932 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:49.116932 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:49.116932 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:40:49.116932 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:40:49.116932 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:40:49.116932 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:40:49.117994 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:40:49.117994 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:40:49.117994 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:40:49.117994 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:40:49.117994 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:49.117994 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:49.117994 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:49.117994 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:49.117994 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:49.117994 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:49.117994 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:49.117994 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:49.117994 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:49.117994 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:49.117994 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:49.117994 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:49.118995 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:49.118995 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:49.118995 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:49.118995 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:49.118995 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:40:49.118995 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:40:49.118995 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:40:49.118995 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:40:49.118995 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:40:49.118995 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:40:49.118995 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:40:49.118995 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:40:49.118995 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 540/1093] 10.222.0.27 () {56 vars in 1096 bytes} [Sun Sep 11 00:40:48 2022] GET /identity/v3/roles/admin => generated 83 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:40:49.124022 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-886db399-c2b3-4d41-845a-29abdfe4793e None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:49.140911 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:49.147358 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799d42a3-4513-4c03-b76f-81fcc289e033 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:49.181536 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a146088-6c41-459c-af4b-5e60a212a9ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:49.182159 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a146088-6c41-459c-af4b-5e60a212a9ba None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:49.182430 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a146088-6c41-459c-af4b-5e60a212a9ba None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:49.182703 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a146088-6c41-459c-af4b-5e60a212a9ba None admin] PATH_INFO: `/v3/users/nova` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:49.189433 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a146088-6c41-459c-af4b-5e60a212a9ba None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:49.190892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a146088-6c41-459c-af4b-5e60a212a9ba None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:49.202944 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-6a146088-6c41-459c-af4b-5e60a212a9ba None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 11 00:40:49.202944 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:49.202944 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:49.202944 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:49.202944 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:49.202944 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:49.202944 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:49.202944 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:49.202944 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:49.202944 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:49.202944 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:49.202944 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:49.202944 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 11 00:40:49.202944 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 11 00:40:49.202944 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 11 00:40:49.202944 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 11 00:40:49.204061 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:49.204061 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:49.204061 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 11 00:40:49.204061 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:40:49.204061 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 11 00:40:49.204061 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:40:49.204061 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:49.204061 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:49.204061 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:49.204061 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:49.204061 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:49.204061 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:49.204061 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:49.204061 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:49.204061 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:49.204061 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:49.204061 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:49.204061 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:49.205031 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:49.205031 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:49.205031 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 11 00:40:49.205031 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 11 00:40:49.205031 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 11 00:40:49.205031 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 11 00:40:49.205031 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 11 00:40:49.205031 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 11 00:40:49.205031 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 11 00:40:49.205031 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:49.205031 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 554/1094] 10.222.0.27 () {56 vars in 1093 bytes} [Sun Sep 11 00:40:49 2022] GET /identity/v3/users/nova => generated 82 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:40:49.209756 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a146088-6c41-459c-af4b-5e60a212a9ba None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:49.227106 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-886db399-c2b3-4d41-845a-29abdfe4793e None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:49.260326 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-027741d9-6cdd-41fc-b970-1d690a2b4145 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:49.260916 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027741d9-6cdd-41fc-b970-1d690a2b4145 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:49.261185 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027741d9-6cdd-41fc-b970-1d690a2b4145 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:49.261449 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027741d9-6cdd-41fc-b970-1d690a2b4145 None admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:49.262100 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-027741d9-6cdd-41fc-b970-1d690a2b4145 None admin] RBAC: Adding query filter params (name=admin) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:49.262393 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-027741d9-6cdd-41fc-b970-1d690a2b4145 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:49.263787 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-027741d9-6cdd-41fc-b970-1d690a2b4145 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:49.265085 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:49.274073 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 541/1095] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:40:49 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:40:49.279992 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-027741d9-6cdd-41fc-b970-1d690a2b4145 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:49.311375 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a146088-6c41-459c-af4b-5e60a212a9ba None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:49.344083 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08943589-08e3-4364-b3b9-6e31f19f15c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:49.344680 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08943589-08e3-4364-b3b9-6e31f19f15c7 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:49.344953 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08943589-08e3-4364-b3b9-6e31f19f15c7 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:49.345217 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08943589-08e3-4364-b3b9-6e31f19f15c7 None admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:49.345966 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08943589-08e3-4364-b3b9-6e31f19f15c7 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:49.346268 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08943589-08e3-4364-b3b9-6e31f19f15c7 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:49.347706 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08943589-08e3-4364-b3b9-6e31f19f15c7 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:49.365664 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 555/1096] 10.222.0.27 () {56 vars in 1133 bytes} [Sun Sep 11 00:40:49 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:40:49.371271 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08943589-08e3-4364-b3b9-6e31f19f15c7 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:49.376937 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-027741d9-6cdd-41fc-b970-1d690a2b4145 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:49.386370 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:49.412282 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0719df52-787d-48b9-a381-365cd751fc42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:49.412904 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0719df52-787d-48b9-a381-365cd751fc42 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:49.413179 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0719df52-787d-48b9-a381-365cd751fc42 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:49.413450 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0719df52-787d-48b9-a381-365cd751fc42 None admin] PATH_INFO: `/v3/users/69253b38ac154e7aad0de65ff9c30949` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:49.414576 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0719df52-787d-48b9-a381-365cd751fc42 None admin] RBAC: Authorizing `identity:get_user(user_id=69253b38ac154e7aad0de65ff9c30949)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:49.416045 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0719df52-787d-48b9-a381-365cd751fc42 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:49.417611 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 542/1097] 10.222.0.27 () {56 vars in 1177 bytes} [Sun Sep 11 00:40:49 2022] GET /identity/v3/users/69253b38ac154e7aad0de65ff9c30949 => generated 285 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:40:49.423250 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0719df52-787d-48b9-a381-365cd751fc42 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:49.469598 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08943589-08e3-4364-b3b9-6e31f19f15c7 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:49.486915 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:49.503866 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca3ee86-d838-4aaf-b2c1-33edc0e54d1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:49.505034 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca3ee86-d838-4aaf-b2c1-33edc0e54d1e None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:49.505303 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca3ee86-d838-4aaf-b2c1-33edc0e54d1e None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:49.505570 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca3ee86-d838-4aaf-b2c1-33edc0e54d1e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:49.524508 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0719df52-787d-48b9-a381-365cd751fc42 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:49.558155 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621806ec-51c3-4888-bb47-b3f099361a60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:49.558753 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621806ec-51c3-4888-bb47-b3f099361a60 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:49.559060 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621806ec-51c3-4888-bb47-b3f099361a60 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:49.559328 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621806ec-51c3-4888-bb47-b3f099361a60 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:49.572461 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca3ee86-d838-4aaf-b2c1-33edc0e54d1e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:49.573915 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca3ee86-d838-4aaf-b2c1-33edc0e54d1e None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:49.588136 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:49.626615 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-621806ec-51c3-4888-bb47-b3f099361a60 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:49.628029 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-621806ec-51c3-4888-bb47-b3f099361a60 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:49.671896 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 556/1098] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:40:49 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 301 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 11 00:40:49.677714 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca3ee86-d838-4aaf-b2c1-33edc0e54d1e None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:49.693376 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:49.731892 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 543/1099] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:40:49 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 11 00:40:49.737478 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621806ec-51c3-4888-bb47-b3f099361a60 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:49.774674 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca3ee86-d838-4aaf-b2c1-33edc0e54d1e None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:49.788551 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:49.808895 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca5fbfa-8bcc-46c2-88d8-47838401840b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:49.809515 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca5fbfa-8bcc-46c2-88d8-47838401840b None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:49.809788 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca5fbfa-8bcc-46c2-88d8-47838401840b None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:49.810053 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca5fbfa-8bcc-46c2-88d8-47838401840b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:49.820400 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca5fbfa-8bcc-46c2-88d8-47838401840b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:49.821823 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca5fbfa-8bcc-46c2-88d8-47838401840b None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:49.834992 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621806ec-51c3-4888-bb47-b3f099361a60 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:49.835863 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-5ca5fbfa-8bcc-46c2-88d8-47838401840b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:40:49.835863 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:49.835863 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:49.835863 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:49.835863 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:49.835863 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:49.835863 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:49.835863 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:49.835863 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:49.835863 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:49.835863 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:49.835863 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:49.835863 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:40:49.835863 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:40:49.835863 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:40:49.835863 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:40:49.836840 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:49.836840 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:49.836840 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:49.836840 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:49.836840 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:49.836840 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:49.836840 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:49.836840 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:49.836840 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:49.836840 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:49.836840 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:49.836840 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:49.836840 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:49.836840 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:49.836840 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:49.836840 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:49.836840 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:40:49.837758 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:40:49.837758 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:40:49.837758 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:49.837758 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 557/1100] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:40:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:40:49.842525 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca5fbfa-8bcc-46c2-88d8-47838401840b None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:49.871308 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14be197-7344-4345-80ae-55829b8b2db9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:49.871932 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14be197-7344-4345-80ae-55829b8b2db9 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:49.872209 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14be197-7344-4345-80ae-55829b8b2db9 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:49.872475 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14be197-7344-4345-80ae-55829b8b2db9 None admin] PATH_INFO: `/v3/projects/b7316adb92cd4d9ba42313ea22a74d3d` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:49.873593 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d14be197-7344-4345-80ae-55829b8b2db9 None admin] RBAC: Authorizing `identity:get_project(project_id=b7316adb92cd4d9ba42313ea22a74d3d)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:49.875046 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d14be197-7344-4345-80ae-55829b8b2db9 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:49.878836 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 544/1101] 10.222.0.27 () {56 vars in 1186 bytes} [Sun Sep 11 00:40:49 2022] GET /identity/v3/projects/b7316adb92cd4d9ba42313ea22a74d3d => generated 311 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:40:49.883240 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14be197-7344-4345-80ae-55829b8b2db9 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:49.914514 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:49.940757 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca5fbfa-8bcc-46c2-88d8-47838401840b None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:49.976303 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2823af5-7d39-4ed1-a8ec-89d343dd3a17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:49.976904 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2823af5-7d39-4ed1-a8ec-89d343dd3a17 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:49.977180 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2823af5-7d39-4ed1-a8ec-89d343dd3a17 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:49.977450 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2823af5-7d39-4ed1-a8ec-89d343dd3a17 None admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:49.978084 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2823af5-7d39-4ed1-a8ec-89d343dd3a17 None admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:49.978380 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2823af5-7d39-4ed1-a8ec-89d343dd3a17 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:49.979778 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2823af5-7d39-4ed1-a8ec-89d343dd3a17 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:49.983957 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14be197-7344-4345-80ae-55829b8b2db9 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:49.992452 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 558/1102] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:40:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:40:49.997988 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2823af5-7d39-4ed1-a8ec-89d343dd3a17 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:50.019543 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc82e73-92e2-464a-a224-bc64f91b9820 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:50.020177 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc82e73-92e2-464a-a224-bc64f91b9820 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:50.020434 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc82e73-92e2-464a-a224-bc64f91b9820 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:50.020700 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc82e73-92e2-464a-a224-bc64f91b9820 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:50.021419 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdc82e73-92e2-464a-a224-bc64f91b9820 None admin] RBAC: Adding query filter params (role.id=7d9213c1187c4b5182bf8132cf23ea19, scope.project.id=b7316adb92cd4d9ba42313ea22a74d3d, user.id=69253b38ac154e7aad0de65ff9c30949) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:50.021725 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdc82e73-92e2-464a-a224-bc64f91b9820 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:50.023175 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdc82e73-92e2-464a-a224-bc64f91b9820 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:50.025161 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:50.028941 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 545/1103] 10.222.0.27 () {56 vars in 1374 bytes} [Sun Sep 11 00:40:49 2022] GET /identity/v3/role_assignments?user.id=69253b38ac154e7aad0de65ff9c30949&scope.project.id=b7316adb92cd4d9ba42313ea22a74d3d&role.id=7d9213c1187c4b5182bf8132cf23ea19 => generated 608 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:40:50.108970 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2823af5-7d39-4ed1-a8ec-89d343dd3a17 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:50.146111 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-349297fd-3504-45ff-8cf0-d979d014c7f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:50.146855 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-349297fd-3504-45ff-8cf0-d979d014c7f9 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:50.147048 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-349297fd-3504-45ff-8cf0-d979d014c7f9 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:50.147318 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-349297fd-3504-45ff-8cf0-d979d014c7f9 None admin] PATH_INFO: `/v3/projects/service` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:50.155226 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-349297fd-3504-45ff-8cf0-d979d014c7f9 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:50.156697 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-349297fd-3504-45ff-8cf0-d979d014c7f9 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:50.169972 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-349297fd-3504-45ff-8cf0-d979d014c7f9 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:40:50.169972 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:40:50.169972 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:40:50.169972 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:40:50.169972 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:40:50.169972 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:40:50.169972 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:40:50.169972 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:40:50.169972 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:40:50.169972 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:40:50.169972 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:40:50.169972 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:40:50.169972 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 11 00:40:50.169972 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 11 00:40:50.169972 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 11 00:40:50.171305 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 11 00:40:50.171305 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:40:50.171305 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:40:50.171305 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:40:50.171305 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:40:50.171305 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:40:50.171305 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:40:50.171305 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:40:50.171305 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:40:50.171305 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:40:50.171305 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:40:50.171305 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:40:50.171305 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:40:50.171305 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:40:50.171305 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:40:50.171305 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:40:50.171305 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:40:50.172238 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 11 00:40:50.172238 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 11 00:40:50.172238 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 11 00:40:50.172238 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 11 00:40:50.172238 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 11 00:40:50.172238 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 11 00:40:50.172238 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:40:50.172238 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:40:50.172238 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 559/1104] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:40:49 2022] GET /identity/v3/projects/service => generated 88 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:40:50.178053 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc82e73-92e2-464a-a224-bc64f91b9820 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:50.276272 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:50.282568 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc82e73-92e2-464a-a224-bc64f91b9820 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:50.316789 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa295495-9ad2-4964-aee8-3830ca5a590a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:50.317395 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa295495-9ad2-4964-aee8-3830ca5a590a None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:50.317670 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa295495-9ad2-4964-aee8-3830ca5a590a None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:50.317939 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa295495-9ad2-4964-aee8-3830ca5a590a None admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:50.318623 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa295495-9ad2-4964-aee8-3830ca5a590a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:50.318959 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa295495-9ad2-4964-aee8-3830ca5a590a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:50.320498 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa295495-9ad2-4964-aee8-3830ca5a590a None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:50.332194 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 546/1105] 10.222.0.27 () {56 vars in 1148 bytes} [Sun Sep 11 00:40:50 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:50.338171 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-349297fd-3504-45ff-8cf0-d979d014c7f9 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:50.422659 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:50.438128 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-349297fd-3504-45ff-8cf0-d979d014c7f9 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:50.471580 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce2bb20-5450-4a36-a036-a6ad87b3defc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:50.472174 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce2bb20-5450-4a36-a036-a6ad87b3defc None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:50.472447 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce2bb20-5450-4a36-a036-a6ad87b3defc None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:50.472713 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce2bb20-5450-4a36-a036-a6ad87b3defc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:50.474339 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce2bb20-5450-4a36-a036-a6ad87b3defc None admin] RBAC: Adding query filter params (role.id=a6b5856c8f5c4144b39cab1b2916ac93, scope.project.id=b504a5058bc6474c8cc077dfeec16799, user.id=835def0e6c6d4365a2767051509eb0d6) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:50.474837 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce2bb20-5450-4a36-a036-a6ad87b3defc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:50.476292 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce2bb20-5450-4a36-a036-a6ad87b3defc None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:50.482374 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 560/1106] 10.222.0.27 () {56 vars in 1374 bytes} [Sun Sep 11 00:40:50 2022] GET /identity/v3/role_assignments?user.id=835def0e6c6d4365a2767051509eb0d6&scope.project.id=b504a5058bc6474c8cc077dfeec16799&role.id=a6b5856c8f5c4144b39cab1b2916ac93 => generated 608 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:52.205273 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab0f193-0aec-4b08-9b35-2603cfd02254 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:52.205843 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab0f193-0aec-4b08-9b35-2603cfd02254 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:52.205843 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab0f193-0aec-4b08-9b35-2603cfd02254 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:52.207788 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 547/1107] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:40:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:40:52.212918 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b94e6a-a0de-468d-96be-1a774d856d64 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:52.213354 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b94e6a-a0de-468d-96be-1a774d856d64 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:52.213459 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b94e6a-a0de-468d-96be-1a774d856d64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:52.231895 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-c6b94e6a-a0de-468d-96be-1a774d856d64 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:40:52.233065 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-c6b94e6a-a0de-468d-96be-1a774d856d64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:40:52.297950 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 561/1108] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:40:52 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:40:52.573238 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bab0f193-0aec-4b08-9b35-2603cfd02254 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:52.671754 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bab0f193-0aec-4b08-9b35-2603cfd02254 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:52.704514 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f60dffc-56e5-4420-b52f-f5cf424a9843 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:52.705119 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f60dffc-56e5-4420-b52f-f5cf424a9843 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:52.705388 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f60dffc-56e5-4420-b52f-f5cf424a9843 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:52.705651 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f60dffc-56e5-4420-b52f-f5cf424a9843 None admin] PATH_INFO: `/v3/domains/default` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:52.706730 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f60dffc-56e5-4420-b52f-f5cf424a9843 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:52.708481 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f60dffc-56e5-4420-b52f-f5cf424a9843 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:52.710000 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 548/1109] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:40:52 2022] GET /identity/v3/domains/default => generated 199 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:52.715549 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6b94e6a-a0de-468d-96be-1a774d856d64 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:52.791866 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:52.811289 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6b94e6a-a0de-468d-96be-1a774d856d64 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:52.843968 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60cbab5e-9f57-490f-871d-31f420402b15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:52.844498 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60cbab5e-9f57-490f-871d-31f420402b15 None admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:52.844769 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60cbab5e-9f57-490f-871d-31f420402b15 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:52.845037 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60cbab5e-9f57-490f-871d-31f420402b15 None admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:52.845763 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60cbab5e-9f57-490f-871d-31f420402b15 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:52.847332 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60cbab5e-9f57-490f-871d-31f420402b15 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:52.868516 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 562/1110] 10.222.0.27 () {60 vars in 1132 bytes} [Sun Sep 11 00:40:52 2022] POST /identity/v3/users => generated 286 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:40:55.060563 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250466f4-3cdd-4bd7-8a17-c720200d1d75 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:55.061142 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250466f4-3cdd-4bd7-8a17-c720200d1d75 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:55.061142 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250466f4-3cdd-4bd7-8a17-c720200d1d75 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:55.062415 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 549/1111] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:40:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:40:55.068771 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f188cfe8-8dc9-48b2-9417-e8cbf918f549 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:55.069192 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f188cfe8-8dc9-48b2-9417-e8cbf918f549 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:55.069301 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f188cfe8-8dc9-48b2-9417-e8cbf918f549 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:55.087361 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-f188cfe8-8dc9-48b2-9417-e8cbf918f549 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:40:55.088542 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-f188cfe8-8dc9-48b2-9417-e8cbf918f549 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:40:55.152948 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 563/1112] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:40:55 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:40:55.420254 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250466f4-3cdd-4bd7-8a17-c720200d1d75 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:55.445577 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:55.523353 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250466f4-3cdd-4bd7-8a17-c720200d1d75 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:55.558510 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85cc000b-0fb3-4b18-aa6f-ee05c4893105 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:55.559156 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85cc000b-0fb3-4b18-aa6f-ee05c4893105 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:55.559435 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85cc000b-0fb3-4b18-aa6f-ee05c4893105 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:55.559707 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85cc000b-0fb3-4b18-aa6f-ee05c4893105 None admin] PATH_INFO: `/v3/roles/518eab0f758241369edcdd997b98ad12` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:55.561089 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85cc000b-0fb3-4b18-aa6f-ee05c4893105 None admin] RBAC: Authorizing `identity:get_role(role_id=518eab0f758241369edcdd997b98ad12)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:55.562462 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85cc000b-0fb3-4b18-aa6f-ee05c4893105 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:55.563805 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 550/1113] 10.222.0.27 () {56 vars in 1177 bytes} [Sun Sep 11 00:40:55 2022] GET /identity/v3/roles/518eab0f758241369edcdd997b98ad12 => generated 223 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:55.570194 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f188cfe8-8dc9-48b2-9417-e8cbf918f549 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:55.633245 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:55.667331 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f188cfe8-8dc9-48b2-9417-e8cbf918f549 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:55.700635 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ebc00a-dc2a-4744-a182-d16fb50f2e0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:55.701202 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ebc00a-dc2a-4744-a182-d16fb50f2e0c None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:55.701481 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ebc00a-dc2a-4744-a182-d16fb50f2e0c None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:55.701752 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ebc00a-dc2a-4744-a182-d16fb50f2e0c None admin] PATH_INFO: `/v3/users/3b27f05b921e404fb2b1d4ee50ce511d` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:55.714971 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43ebc00a-dc2a-4744-a182-d16fb50f2e0c None admin] RBAC: Authorizing `identity:get_user(user_id=3b27f05b921e404fb2b1d4ee50ce511d)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:55.716392 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43ebc00a-dc2a-4744-a182-d16fb50f2e0c None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:55.717968 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 564/1114] 10.222.0.27 () {56 vars in 1177 bytes} [Sun Sep 11 00:40:55 2022] GET /identity/v3/users/3b27f05b921e404fb2b1d4ee50ce511d => generated 286 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:55.724282 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85cc000b-0fb3-4b18-aa6f-ee05c4893105 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:55.811830 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:55.820876 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85cc000b-0fb3-4b18-aa6f-ee05c4893105 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:55.854412 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45af6872-7f92-4f4d-9495-3743023e3e02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:55.855023 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45af6872-7f92-4f4d-9495-3743023e3e02 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:55.855298 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45af6872-7f92-4f4d-9495-3743023e3e02 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:55.855568 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45af6872-7f92-4f4d-9495-3743023e3e02 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:55.920125 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45af6872-7f92-4f4d-9495-3743023e3e02 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:55.921554 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45af6872-7f92-4f4d-9495-3743023e3e02 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:56.011036 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:56.021049 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 551/1115] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:40:55 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 298 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 11 00:40:56.027021 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ebc00a-dc2a-4744-a182-d16fb50f2e0c None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:56.123850 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ebc00a-dc2a-4744-a182-d16fb50f2e0c None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:56.147033 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:56.157790 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-849afdb7-54c0-45d8-81c2-fe8cb2815b98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:56.158413 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849afdb7-54c0-45d8-81c2-fe8cb2815b98 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:56.158691 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849afdb7-54c0-45d8-81c2-fe8cb2815b98 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:56.158996 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849afdb7-54c0-45d8-81c2-fe8cb2815b98 None admin] PATH_INFO: `/v3/projects/b7316adb92cd4d9ba42313ea22a74d3d` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:56.160383 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-849afdb7-54c0-45d8-81c2-fe8cb2815b98 None admin] RBAC: Authorizing `identity:get_project(project_id=b7316adb92cd4d9ba42313ea22a74d3d)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:56.161905 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-849afdb7-54c0-45d8-81c2-fe8cb2815b98 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:56.163558 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 565/1116] 10.222.0.27 () {56 vars in 1186 bytes} [Sun Sep 11 00:40:56 2022] GET /identity/v3/projects/b7316adb92cd4d9ba42313ea22a74d3d => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:56.169861 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45af6872-7f92-4f4d-9495-3743023e3e02 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:56.265136 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45af6872-7f92-4f4d-9495-3743023e3e02 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:56.298488 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f94f91e7-c62f-4cbe-a3e1-2e6f77d368f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:56.299095 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f94f91e7-c62f-4cbe-a3e1-2e6f77d368f3 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:56.299370 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f94f91e7-c62f-4cbe-a3e1-2e6f77d368f3 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:56.299633 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f94f91e7-c62f-4cbe-a3e1-2e6f77d368f3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:56.300342 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f94f91e7-c62f-4cbe-a3e1-2e6f77d368f3 None admin] RBAC: Adding query filter params (role.id=518eab0f758241369edcdd997b98ad12, scope.project.id=b7316adb92cd4d9ba42313ea22a74d3d, user.id=3b27f05b921e404fb2b1d4ee50ce511d) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:40:56.300645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f94f91e7-c62f-4cbe-a3e1-2e6f77d368f3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:56.302751 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f94f91e7-c62f-4cbe-a3e1-2e6f77d368f3 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:56.307996 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 552/1117] 10.222.0.27 () {56 vars in 1374 bytes} [Sun Sep 11 00:40:56 2022] GET /identity/v3/role_assignments?user.id=3b27f05b921e404fb2b1d4ee50ce511d&scope.project.id=b7316adb92cd4d9ba42313ea22a74d3d&role.id=518eab0f758241369edcdd997b98ad12 => generated 259 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:58.412611 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9823f1-869f-4338-9e58-7a54a1108fda None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:58.413132 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9823f1-869f-4338-9e58-7a54a1108fda None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:58.413132 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9823f1-869f-4338-9e58-7a54a1108fda None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:58.414409 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 566/1118] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:40:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:40:58.419606 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b258f479-52ec-4801-94c8-1cd7cac9e04e None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:58.419992 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b258f479-52ec-4801-94c8-1cd7cac9e04e None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:58.420141 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b258f479-52ec-4801-94c8-1cd7cac9e04e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:58.438413 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-b258f479-52ec-4801-94c8-1cd7cac9e04e None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:40:58.439616 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-b258f479-52ec-4801-94c8-1cd7cac9e04e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:40:58.487372 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:58.504689 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 553/1119] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:40:58 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:40:58.768560 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9823f1-869f-4338-9e58-7a54a1108fda None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:58.867761 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9823f1-869f-4338-9e58-7a54a1108fda None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:58.900312 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2ec337-feb5-4e22-87ea-94cb3f31d361 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:58.900923 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2ec337-feb5-4e22-87ea-94cb3f31d361 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:58.901198 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2ec337-feb5-4e22-87ea-94cb3f31d361 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:58.901463 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2ec337-feb5-4e22-87ea-94cb3f31d361 None admin] PATH_INFO: `/v3/roles/518eab0f758241369edcdd997b98ad12` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:58.902847 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b2ec337-feb5-4e22-87ea-94cb3f31d361 None admin] RBAC: Authorizing `identity:get_role(role_id=518eab0f758241369edcdd997b98ad12)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:58.904198 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b2ec337-feb5-4e22-87ea-94cb3f31d361 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:58.905487 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 567/1120] 10.222.0.27 () {56 vars in 1177 bytes} [Sun Sep 11 00:40:58 2022] GET /identity/v3/roles/518eab0f758241369edcdd997b98ad12 => generated 223 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:58.912146 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b258f479-52ec-4801-94c8-1cd7cac9e04e None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:59.006889 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b258f479-52ec-4801-94c8-1cd7cac9e04e None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:59.040696 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d590d99-c80a-4eff-8b29-f80f8586cc6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:59.041255 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d590d99-c80a-4eff-8b29-f80f8586cc6e None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:59.041531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d590d99-c80a-4eff-8b29-f80f8586cc6e None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:59.041799 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d590d99-c80a-4eff-8b29-f80f8586cc6e None admin] PATH_INFO: `/v3/users/3b27f05b921e404fb2b1d4ee50ce511d` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:59.042940 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d590d99-c80a-4eff-8b29-f80f8586cc6e None admin] RBAC: Authorizing `identity:get_user(user_id=3b27f05b921e404fb2b1d4ee50ce511d)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:59.044362 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d590d99-c80a-4eff-8b29-f80f8586cc6e None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:59.045898 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 554/1121] 10.222.0.27 () {56 vars in 1177 bytes} [Sun Sep 11 00:40:58 2022] GET /identity/v3/users/3b27f05b921e404fb2b1d4ee50ce511d => generated 286 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:59.051738 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2ec337-feb5-4e22-87ea-94cb3f31d361 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:59.110302 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:59.151117 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2ec337-feb5-4e22-87ea-94cb3f31d361 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:59.183273 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95015c7f-806f-4bca-84de-d4cfaab26cc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:59.183896 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95015c7f-806f-4bca-84de-d4cfaab26cc0 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:59.184171 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95015c7f-806f-4bca-84de-d4cfaab26cc0 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:59.184441 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95015c7f-806f-4bca-84de-d4cfaab26cc0 None admin] PATH_INFO: `/v3/projects/b7316adb92cd4d9ba42313ea22a74d3d` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:59.185543 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95015c7f-806f-4bca-84de-d4cfaab26cc0 None admin] RBAC: Authorizing `identity:get_project(project_id=b7316adb92cd4d9ba42313ea22a74d3d)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:59.187004 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95015c7f-806f-4bca-84de-d4cfaab26cc0 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:59.188613 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 568/1122] 10.222.0.27 () {56 vars in 1186 bytes} [Sun Sep 11 00:40:59 2022] GET /identity/v3/projects/b7316adb92cd4d9ba42313ea22a74d3d => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:40:59.195314 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d590d99-c80a-4eff-8b29-f80f8586cc6e None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:40:59.248847 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:40:59.292869 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d590d99-c80a-4eff-8b29-f80f8586cc6e None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:40:59.325603 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee359b0e-875f-4481-b978-108b47788953 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:40:59.326240 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee359b0e-875f-4481-b978-108b47788953 None admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:40:59.326516 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee359b0e-875f-4481-b978-108b47788953 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:40:59.326811 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee359b0e-875f-4481-b978-108b47788953 None admin] PATH_INFO: `/v3/projects/b7316adb92cd4d9ba42313ea22a74d3d/users/3b27f05b921e404fb2b1d4ee50ce511d/roles/518eab0f758241369edcdd997b98ad12` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:40:59.328921 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee359b0e-875f-4481-b978-108b47788953 None admin] RBAC: Authorizing `identity:create_grant(project_id=b7316adb92cd4d9ba42313ea22a74d3d, user_id=3b27f05b921e404fb2b1d4ee50ce511d, role_id=518eab0f758241369edcdd997b98ad12)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:40:59.330340 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee359b0e-875f-4481-b978-108b47788953 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:40:59.342866 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 555/1123] 10.222.0.27 () {58 vars in 1439 bytes} [Sun Sep 11 00:40:59 2022] PUT /identity/v3/projects/b7316adb92cd4d9ba42313ea22a74d3d/users/3b27f05b921e404fb2b1d4ee50ce511d/roles/518eab0f758241369edcdd997b98ad12 => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:41:01.451063 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddce87be-db9d-4ef4-9c52-fcc854e656de None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:01.451584 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddce87be-db9d-4ef4-9c52-fcc854e656de None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:01.451584 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddce87be-db9d-4ef4-9c52-fcc854e656de None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:01.452876 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 569/1124] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:41:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:41:01.458751 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2a1118-f2c3-497b-b3ed-4206c58132ad None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:01.459190 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2a1118-f2c3-497b-b3ed-4206c58132ad None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:01.459308 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2a1118-f2c3-497b-b3ed-4206c58132ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:01.476919 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-5f2a1118-f2c3-497b-b3ed-4206c58132ad None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:41:01.478075 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-5f2a1118-f2c3-497b-b3ed-4206c58132ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:41:01.542388 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 556/1125] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:41:01 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:41:01.809900 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddce87be-db9d-4ef4-9c52-fcc854e656de None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:41:01.907662 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddce87be-db9d-4ef4-9c52-fcc854e656de None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:41:01.933731 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:41:01.942296 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f72c27b-82b8-4cd5-be60-92e1deec3a14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:41:01.942951 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f72c27b-82b8-4cd5-be60-92e1deec3a14 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:01.943228 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f72c27b-82b8-4cd5-be60-92e1deec3a14 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:01.943494 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f72c27b-82b8-4cd5-be60-92e1deec3a14 None admin] PATH_INFO: `/v3/roles/518eab0f758241369edcdd997b98ad12` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:01.944865 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f72c27b-82b8-4cd5-be60-92e1deec3a14 None admin] RBAC: Authorizing `identity:get_role(role_id=518eab0f758241369edcdd997b98ad12)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:41:01.946220 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f72c27b-82b8-4cd5-be60-92e1deec3a14 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:41:01.947562 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 570/1126] 10.222.0.27 () {56 vars in 1177 bytes} [Sun Sep 11 00:41:01 2022] GET /identity/v3/roles/518eab0f758241369edcdd997b98ad12 => generated 223 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:41:01.953324 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f2a1118-f2c3-497b-b3ed-4206c58132ad None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:41:01.999197 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:41:02.053819 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f2a1118-f2c3-497b-b3ed-4206c58132ad None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:41:02.086770 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d3e8a2-ec13-4338-bb3b-44f73d5fd08b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:41:02.087358 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d3e8a2-ec13-4338-bb3b-44f73d5fd08b None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:02.087632 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d3e8a2-ec13-4338-bb3b-44f73d5fd08b None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:02.087902 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d3e8a2-ec13-4338-bb3b-44f73d5fd08b None admin] PATH_INFO: `/v3/users/3b27f05b921e404fb2b1d4ee50ce511d` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:02.089015 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d3e8a2-ec13-4338-bb3b-44f73d5fd08b None admin] RBAC: Authorizing `identity:get_user(user_id=3b27f05b921e404fb2b1d4ee50ce511d)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:41:02.090448 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d3e8a2-ec13-4338-bb3b-44f73d5fd08b None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:41:02.092047 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 557/1127] 10.222.0.27 () {56 vars in 1177 bytes} [Sun Sep 11 00:41:01 2022] GET /identity/v3/users/3b27f05b921e404fb2b1d4ee50ce511d => generated 286 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:41:02.098337 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f72c27b-82b8-4cd5-be60-92e1deec3a14 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:41:02.195454 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f72c27b-82b8-4cd5-be60-92e1deec3a14 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:41:02.228222 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4865b7c4-cfc5-4b5d-92e5-253308c0af76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:41:02.228821 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4865b7c4-cfc5-4b5d-92e5-253308c0af76 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:02.229104 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4865b7c4-cfc5-4b5d-92e5-253308c0af76 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:02.229375 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4865b7c4-cfc5-4b5d-92e5-253308c0af76 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:02.288151 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:41:02.295016 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4865b7c4-cfc5-4b5d-92e5-253308c0af76 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:41:02.296456 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4865b7c4-cfc5-4b5d-92e5-253308c0af76 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:41:02.390722 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 571/1128] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:41:02 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 11 00:41:02.396604 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d3e8a2-ec13-4338-bb3b-44f73d5fd08b None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:41:02.494111 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d3e8a2-ec13-4338-bb3b-44f73d5fd08b None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:41:02.506766 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:41:02.527765 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250a21df-8e17-4ce5-8c44-713bda38c818 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:41:02.528382 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250a21df-8e17-4ce5-8c44-713bda38c818 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:02.528657 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250a21df-8e17-4ce5-8c44-713bda38c818 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:02.528927 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250a21df-8e17-4ce5-8c44-713bda38c818 None admin] PATH_INFO: `/v3/projects/b7316adb92cd4d9ba42313ea22a74d3d` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:02.530023 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-250a21df-8e17-4ce5-8c44-713bda38c818 None admin] RBAC: Authorizing `identity:get_project(project_id=b7316adb92cd4d9ba42313ea22a74d3d)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:41:02.531484 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-250a21df-8e17-4ce5-8c44-713bda38c818 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:41:02.533096 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 558/1129] 10.222.0.27 () {56 vars in 1186 bytes} [Sun Sep 11 00:41:02 2022] GET /identity/v3/projects/b7316adb92cd4d9ba42313ea22a74d3d => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:41:02.540120 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4865b7c4-cfc5-4b5d-92e5-253308c0af76 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:41:02.630371 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:41:02.636415 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4865b7c4-cfc5-4b5d-92e5-253308c0af76 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:41:02.670011 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c13e4f-262e-45d5-a4f5-753967456f33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:41:02.670555 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c13e4f-262e-45d5-a4f5-753967456f33 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:02.670859 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c13e4f-262e-45d5-a4f5-753967456f33 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:02.671132 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c13e4f-262e-45d5-a4f5-753967456f33 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:02.671847 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71c13e4f-262e-45d5-a4f5-753967456f33 None admin] RBAC: Adding query filter params (role.id=518eab0f758241369edcdd997b98ad12, scope.project.id=b7316adb92cd4d9ba42313ea22a74d3d, user.id=3b27f05b921e404fb2b1d4ee50ce511d) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:41:02.672155 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71c13e4f-262e-45d5-a4f5-753967456f33 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:41:02.673577 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71c13e4f-262e-45d5-a4f5-753967456f33 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:41:02.679166 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 572/1130] 10.222.0.27 () {56 vars in 1374 bytes} [Sun Sep 11 00:41:02 2022] GET /identity/v3/role_assignments?user.id=3b27f05b921e404fb2b1d4ee50ce511d&scope.project.id=b7316adb92cd4d9ba42313ea22a74d3d&role.id=518eab0f758241369edcdd997b98ad12 => generated 608 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:41:04.820667 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17cece75-7fb3-4fee-8c18-8e9228a3355f None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:04.821276 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17cece75-7fb3-4fee-8c18-8e9228a3355f None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:04.821276 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17cece75-7fb3-4fee-8c18-8e9228a3355f None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:04.822543 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 559/1131] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:41:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:41:04.827672 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6994c93-cc20-4cab-a14a-e2c0cc12b3da None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:04.828047 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6994c93-cc20-4cab-a14a-e2c0cc12b3da None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:04.828219 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6994c93-cc20-4cab-a14a-e2c0cc12b3da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:04.846318 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-b6994c93-cc20-4cab-a14a-e2c0cc12b3da None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:41:04.848103 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-b6994c93-cc20-4cab-a14a-e2c0cc12b3da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:41:04.913516 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 573/1132] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:41:04 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:41:05.187400 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17cece75-7fb3-4fee-8c18-8e9228a3355f None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:41:05.286755 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17cece75-7fb3-4fee-8c18-8e9228a3355f None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:41:05.325023 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba5d9528-925c-47e2-b070-6a23bbd936b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:41:05.325902 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5d9528-925c-47e2-b070-6a23bbd936b5 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:05.326278 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5d9528-925c-47e2-b070-6a23bbd936b5 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:05.326645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5d9528-925c-47e2-b070-6a23bbd936b5 None admin] PATH_INFO: `/v3/domains/default` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:05.328329 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba5d9528-925c-47e2-b070-6a23bbd936b5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:41:05.329811 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba5d9528-925c-47e2-b070-6a23bbd936b5 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:41:05.331479 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 560/1133] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:41:05 2022] GET /identity/v3/domains/default => generated 199 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:41:05.337347 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6994c93-cc20-4cab-a14a-e2c0cc12b3da None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:41:05.424218 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:41:05.434580 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6994c93-cc20-4cab-a14a-e2c0cc12b3da None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:41:05.468326 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-550f9b93-1524-440d-a5bf-b29e602d96ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:41:05.468920 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550f9b93-1524-440d-a5bf-b29e602d96ba None admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:05.469194 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550f9b93-1524-440d-a5bf-b29e602d96ba None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:05.469485 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550f9b93-1524-440d-a5bf-b29e602d96ba None admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:05.470122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-550f9b93-1524-440d-a5bf-b29e602d96ba None admin] RBAC: Authorizing `identity:create_project()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:41:05.471561 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-550f9b93-1524-440d-a5bf-b29e602d96ba None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:41:05.498961 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 574/1134] 10.222.0.27 () {60 vars in 1141 bytes} [Sun Sep 11 00:41:05 2022] POST /identity/v3/projects => generated 311 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:41:07.682429 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ee734f-8f1e-4334-b97e-07387b6e948e None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:07.683036 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ee734f-8f1e-4334-b97e-07387b6e948e None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:07.683036 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ee734f-8f1e-4334-b97e-07387b6e948e None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:07.684334 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 561/1135] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:41:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:41:07.689535 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59be4bd-a56d-4321-9811-8cd09af7bf6d None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:07.689915 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59be4bd-a56d-4321-9811-8cd09af7bf6d None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:07.690079 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59be4bd-a56d-4321-9811-8cd09af7bf6d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:07.708873 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-c59be4bd-a56d-4321-9811-8cd09af7bf6d None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:41:07.710025 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-c59be4bd-a56d-4321-9811-8cd09af7bf6d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:41:07.774213 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 575/1136] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:41:07 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 11 00:41:08.043582 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0ee734f-8f1e-4334-b97e-07387b6e948e None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:41:08.094037 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:41:08.145763 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0ee734f-8f1e-4334-b97e-07387b6e948e None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:41:08.179103 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7935a45f-a0b7-42bc-86b6-12cd4128e220 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:41:08.179714 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7935a45f-a0b7-42bc-86b6-12cd4128e220 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:08.179989 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7935a45f-a0b7-42bc-86b6-12cd4128e220 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:08.180254 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7935a45f-a0b7-42bc-86b6-12cd4128e220 None admin] PATH_INFO: `/v3/domains/default` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:08.181339 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7935a45f-a0b7-42bc-86b6-12cd4128e220 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:41:08.182756 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7935a45f-a0b7-42bc-86b6-12cd4128e220 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:41:08.184321 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 562/1137] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:41:08 2022] GET /identity/v3/domains/default => generated 199 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:41:08.189970 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c59be4bd-a56d-4321-9811-8cd09af7bf6d None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:41:08.256026 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:41:08.284092 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c59be4bd-a56d-4321-9811-8cd09af7bf6d None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:41:08.317585 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a8ed5da-42dd-42c8-aa14-26780a05f4d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:41:08.318118 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8ed5da-42dd-42c8-aa14-26780a05f4d0 None admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:08.318390 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8ed5da-42dd-42c8-aa14-26780a05f4d0 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:08.318654 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8ed5da-42dd-42c8-aa14-26780a05f4d0 None admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:08.319332 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a8ed5da-42dd-42c8-aa14-26780a05f4d0 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:41:08.320759 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a8ed5da-42dd-42c8-aa14-26780a05f4d0 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:41:08.341271 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 576/1138] 10.222.0.27 () {60 vars in 1132 bytes} [Sun Sep 11 00:41:08 2022] POST /identity/v3/users => generated 286 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:41:10.521329 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8776fa19-188a-4c6d-81bc-59f883471b2c None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:10.521869 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8776fa19-188a-4c6d-81bc-59f883471b2c None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:10.521869 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8776fa19-188a-4c6d-81bc-59f883471b2c None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:10.523221 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 563/1139] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:41:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 11 00:41:10.528677 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f6ff03-f8c2-4197-985a-af0ec3a5cc42 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:10.529124 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f6ff03-f8c2-4197-985a-af0ec3a5cc42 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:10.529220 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f6ff03-f8c2-4197-985a-af0ec3a5cc42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:10.547236 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-22f6ff03-f8c2-4197-985a-af0ec3a5cc42 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:41:10.548404 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-22f6ff03-f8c2-4197-985a-af0ec3a5cc42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:41:10.612339 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 577/1140] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:41:10 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:41:10.880892 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8776fa19-188a-4c6d-81bc-59f883471b2c None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:41:10.979046 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8776fa19-188a-4c6d-81bc-59f883471b2c None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:41:10.996393 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:41:11.015382 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-675a23b4-411e-4c3b-bb16-7f19bade4f3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:41:11.015999 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-675a23b4-411e-4c3b-bb16-7f19bade4f3f None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:11.016272 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-675a23b4-411e-4c3b-bb16-7f19bade4f3f None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:11.016538 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-675a23b4-411e-4c3b-bb16-7f19bade4f3f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:11.026240 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-675a23b4-411e-4c3b-bb16-7f19bade4f3f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:41:11.031154 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-675a23b4-411e-4c3b-bb16-7f19bade4f3f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:41:11.031154 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:41:11.031154 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:41:11.031154 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:41:11.031154 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:41:11.031154 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:41:11.031154 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:41:11.031154 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:41:11.031154 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:41:11.031154 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:41:11.031154 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:41:11.031154 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:41:11.031154 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:41:11.031154 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:41:11.031154 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:41:11.031154 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:41:11.032267 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:41:11.032267 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:41:11.032267 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:41:11.032267 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:41:11.032267 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:41:11.032267 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:41:11.032267 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:41:11.032267 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:41:11.032267 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:41:11.032267 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:41:11.032267 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:41:11.032267 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:41:11.032267 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:41:11.032267 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:41:11.032267 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:41:11.032267 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:41:11.033480 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:41:11.033480 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:41:11.033480 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:41:11.033480 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:41:11.033480 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:41:11.033480 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:41:11.033480 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:41:11.033480 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:41:11.033480 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:41:11.033480 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:41:11.033480 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:41:11.033480 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:41:11.033480 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 564/1141] 10.222.0.27 () {56 vars in 1096 bytes} [Sun Sep 11 00:41:10 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:41:11.038637 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22f6ff03-f8c2-4197-985a-af0ec3a5cc42 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:41:11.094566 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:41:11.136054 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22f6ff03-f8c2-4197-985a-af0ec3a5cc42 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:41:11.168197 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1844dcf-c42c-4c05-b40d-1fc45178c0db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:41:11.168796 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1844dcf-c42c-4c05-b40d-1fc45178c0db None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:11.169073 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1844dcf-c42c-4c05-b40d-1fc45178c0db None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:11.169345 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1844dcf-c42c-4c05-b40d-1fc45178c0db None admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:11.169998 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1844dcf-c42c-4c05-b40d-1fc45178c0db None admin] RBAC: Adding query filter params (name=admin) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:41:11.170297 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1844dcf-c42c-4c05-b40d-1fc45178c0db None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:41:11.171685 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1844dcf-c42c-4c05-b40d-1fc45178c0db None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:41:11.180029 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 578/1142] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:41:11 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:41:11.185967 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-675a23b4-411e-4c3b-bb16-7f19bade4f3f None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:41:11.281875 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-675a23b4-411e-4c3b-bb16-7f19bade4f3f None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:41:11.316059 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e137ec8b-f5de-4f0f-bbbf-2e1c90f08dfb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:41:11.316832 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e137ec8b-f5de-4f0f-bbbf-2e1c90f08dfb None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:11.317107 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e137ec8b-f5de-4f0f-bbbf-2e1c90f08dfb None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:11.317377 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e137ec8b-f5de-4f0f-bbbf-2e1c90f08dfb None admin] PATH_INFO: `/v3/users/bdf9d856e7904c1187f1803ef0f9ae93` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:11.329952 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e137ec8b-f5de-4f0f-bbbf-2e1c90f08dfb None admin] RBAC: Authorizing `identity:get_user(user_id=bdf9d856e7904c1187f1803ef0f9ae93)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:41:11.331462 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e137ec8b-f5de-4f0f-bbbf-2e1c90f08dfb None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:41:11.333170 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 565/1143] 10.222.0.27 () {56 vars in 1177 bytes} [Sun Sep 11 00:41:11 2022] GET /identity/v3/users/bdf9d856e7904c1187f1803ef0f9ae93 => generated 286 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:41:11.338903 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1844dcf-c42c-4c05-b40d-1fc45178c0db None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:41:11.435902 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1844dcf-c42c-4c05-b40d-1fc45178c0db None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:41:11.454356 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:41:11.468556 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b3236e-385d-47c3-be83-71b5fc66a99f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:41:11.469150 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b3236e-385d-47c3-be83-71b5fc66a99f None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:11.469447 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b3236e-385d-47c3-be83-71b5fc66a99f None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:11.469713 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b3236e-385d-47c3-be83-71b5fc66a99f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:11.533768 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6b3236e-385d-47c3-be83-71b5fc66a99f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:41:11.535518 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6b3236e-385d-47c3-be83-71b5fc66a99f None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:41:11.633375 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 579/1144] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:41:11 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 295 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 11 00:41:11.639280 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e137ec8b-f5de-4f0f-bbbf-2e1c90f08dfb None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:41:11.673537 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:41:11.738977 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e137ec8b-f5de-4f0f-bbbf-2e1c90f08dfb None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:41:11.771897 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db139fb9-2568-48c3-ab09-31b779dab251 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:41:11.772504 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db139fb9-2568-48c3-ab09-31b779dab251 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:11.772778 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db139fb9-2568-48c3-ab09-31b779dab251 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:11.773045 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db139fb9-2568-48c3-ab09-31b779dab251 None admin] PATH_INFO: `/v3/projects/9a9b03da26eb42aa9c9816873d8afb74` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:11.774136 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db139fb9-2568-48c3-ab09-31b779dab251 None admin] RBAC: Authorizing `identity:get_project(project_id=9a9b03da26eb42aa9c9816873d8afb74)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:41:11.775573 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db139fb9-2568-48c3-ab09-31b779dab251 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:41:11.777151 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 566/1145] 10.222.0.27 () {56 vars in 1186 bytes} [Sun Sep 11 00:41:11 2022] GET /identity/v3/projects/9a9b03da26eb42aa9c9816873d8afb74 => generated 311 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:41:11.783238 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b3236e-385d-47c3-be83-71b5fc66a99f None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:41:11.804355 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:41:11.880610 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b3236e-385d-47c3-be83-71b5fc66a99f None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:41:11.912920 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63305756-6fd8-43e6-b46e-aac06e96edc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:41:11.913510 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63305756-6fd8-43e6-b46e-aac06e96edc1 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:11.913786 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63305756-6fd8-43e6-b46e-aac06e96edc1 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:11.914052 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63305756-6fd8-43e6-b46e-aac06e96edc1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:11.914759 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63305756-6fd8-43e6-b46e-aac06e96edc1 None admin] RBAC: Adding query filter params (role.id=7d9213c1187c4b5182bf8132cf23ea19, scope.project.id=9a9b03da26eb42aa9c9816873d8afb74, user.id=bdf9d856e7904c1187f1803ef0f9ae93) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:41:11.915094 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63305756-6fd8-43e6-b46e-aac06e96edc1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:41:11.916507 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63305756-6fd8-43e6-b46e-aac06e96edc1 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:41:11.921673 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 580/1146] 10.222.0.27 () {56 vars in 1374 bytes} [Sun Sep 11 00:41:11 2022] GET /identity/v3/role_assignments?user.id=bdf9d856e7904c1187f1803ef0f9ae93&scope.project.id=9a9b03da26eb42aa9c9816873d8afb74&role.id=7d9213c1187c4b5182bf8132cf23ea19 => generated 259 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:41:14.047930 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860b1c06-44b8-41d8-8ce2-24eb27b35595 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:14.048470 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860b1c06-44b8-41d8-8ce2-24eb27b35595 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:14.048470 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860b1c06-44b8-41d8-8ce2-24eb27b35595 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:14.049805 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 567/1147] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:41:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:41:14.055973 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be25de6-9d90-40f0-ad4c-34a6b868aac6 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:14.056387 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be25de6-9d90-40f0-ad4c-34a6b868aac6 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:14.056521 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be25de6-9d90-40f0-ad4c-34a6b868aac6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:14.074760 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-9be25de6-9d90-40f0-ad4c-34a6b868aac6 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:41:14.075967 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-9be25de6-9d90-40f0-ad4c-34a6b868aac6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:41:14.139857 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 581/1148] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:41:14 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:41:14.408678 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-860b1c06-44b8-41d8-8ce2-24eb27b35595 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:41:14.507124 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-860b1c06-44b8-41d8-8ce2-24eb27b35595 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:41:14.508595 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:41:14.541646 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-060ac4c8-9fcf-497a-b9f2-51ef678fc2d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:41:14.542277 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-060ac4c8-9fcf-497a-b9f2-51ef678fc2d2 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:14.542553 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-060ac4c8-9fcf-497a-b9f2-51ef678fc2d2 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:14.542844 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-060ac4c8-9fcf-497a-b9f2-51ef678fc2d2 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:14.553232 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-060ac4c8-9fcf-497a-b9f2-51ef678fc2d2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:41:14.558093 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-060ac4c8-9fcf-497a-b9f2-51ef678fc2d2 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:41:14.558093 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:41:14.558093 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:41:14.558093 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:41:14.558093 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:41:14.558093 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:41:14.558093 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:41:14.558093 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:41:14.558093 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:41:14.558093 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:41:14.558093 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:41:14.558093 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:41:14.558093 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:41:14.558093 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:41:14.558093 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:41:14.558093 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:41:14.559216 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:41:14.559216 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:41:14.559216 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:41:14.559216 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:41:14.559216 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:41:14.559216 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:41:14.559216 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:41:14.559216 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:41:14.559216 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:41:14.559216 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:41:14.559216 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:41:14.559216 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:41:14.559216 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:41:14.559216 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:41:14.559216 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:41:14.559216 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:41:14.560265 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:41:14.560265 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:41:14.560265 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:41:14.560265 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:41:14.560265 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:41:14.560265 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:41:14.560265 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:41:14.560265 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:41:14.560265 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:41:14.560265 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:41:14.560265 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:41:14.560265 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:41:14.560265 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 568/1149] 10.222.0.27 () {56 vars in 1096 bytes} [Sun Sep 11 00:41:14 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:41:14.565236 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be25de6-9d90-40f0-ad4c-34a6b868aac6 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:41:14.580139 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:41:14.668954 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be25de6-9d90-40f0-ad4c-34a6b868aac6 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:41:14.701581 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03277a4b-0a1d-4c40-af34-7c96fbff6976 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:41:14.702176 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03277a4b-0a1d-4c40-af34-7c96fbff6976 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:14.702455 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03277a4b-0a1d-4c40-af34-7c96fbff6976 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:14.702724 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03277a4b-0a1d-4c40-af34-7c96fbff6976 None admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:14.703411 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03277a4b-0a1d-4c40-af34-7c96fbff6976 None admin] RBAC: Adding query filter params (name=admin) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:41:14.703713 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03277a4b-0a1d-4c40-af34-7c96fbff6976 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:41:14.705067 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03277a4b-0a1d-4c40-af34-7c96fbff6976 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:41:14.713419 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 582/1150] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:41:14 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:41:14.719234 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-060ac4c8-9fcf-497a-b9f2-51ef678fc2d2 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:41:14.818510 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-060ac4c8-9fcf-497a-b9f2-51ef678fc2d2 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:41:14.852037 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a55ab593-5c7c-4118-9f24-b71ad39e47f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:41:14.852655 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a55ab593-5c7c-4118-9f24-b71ad39e47f8 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:14.852933 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a55ab593-5c7c-4118-9f24-b71ad39e47f8 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:14.853206 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a55ab593-5c7c-4118-9f24-b71ad39e47f8 None admin] PATH_INFO: `/v3/users/bdf9d856e7904c1187f1803ef0f9ae93` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:14.854309 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a55ab593-5c7c-4118-9f24-b71ad39e47f8 None admin] RBAC: Authorizing `identity:get_user(user_id=bdf9d856e7904c1187f1803ef0f9ae93)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:41:14.855763 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a55ab593-5c7c-4118-9f24-b71ad39e47f8 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:41:14.857356 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 569/1151] 10.222.0.27 () {56 vars in 1177 bytes} [Sun Sep 11 00:41:14 2022] GET /identity/v3/users/bdf9d856e7904c1187f1803ef0f9ae93 => generated 286 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:41:14.863269 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03277a4b-0a1d-4c40-af34-7c96fbff6976 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:41:14.937504 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:41:14.960390 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03277a4b-0a1d-4c40-af34-7c96fbff6976 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:41:14.992638 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b6c9d3-d087-4101-8e3e-2bf50d18947a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:41:14.993254 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b6c9d3-d087-4101-8e3e-2bf50d18947a None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:14.993529 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b6c9d3-d087-4101-8e3e-2bf50d18947a None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:14.993797 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b6c9d3-d087-4101-8e3e-2bf50d18947a None admin] PATH_INFO: `/v3/projects/9a9b03da26eb42aa9c9816873d8afb74` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:14.994977 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20b6c9d3-d087-4101-8e3e-2bf50d18947a None admin] RBAC: Authorizing `identity:get_project(project_id=9a9b03da26eb42aa9c9816873d8afb74)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:41:14.998511 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20b6c9d3-d087-4101-8e3e-2bf50d18947a None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:41:15.001302 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 583/1152] 10.222.0.27 () {56 vars in 1186 bytes} [Sun Sep 11 00:41:14 2022] GET /identity/v3/projects/9a9b03da26eb42aa9c9816873d8afb74 => generated 311 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:41:15.007894 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a55ab593-5c7c-4118-9f24-b71ad39e47f8 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:41:15.037939 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:41:15.106453 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a55ab593-5c7c-4118-9f24-b71ad39e47f8 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:41:15.139114 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e39894de-7043-4095-87e8-e6a3ec0a462f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:41:15.139748 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39894de-7043-4095-87e8-e6a3ec0a462f None admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:15.140019 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39894de-7043-4095-87e8-e6a3ec0a462f None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:15.140283 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39894de-7043-4095-87e8-e6a3ec0a462f None admin] PATH_INFO: `/v3/projects/9a9b03da26eb42aa9c9816873d8afb74/users/bdf9d856e7904c1187f1803ef0f9ae93/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:15.142132 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e39894de-7043-4095-87e8-e6a3ec0a462f None admin] RBAC: Authorizing `identity:create_grant(project_id=9a9b03da26eb42aa9c9816873d8afb74, user_id=bdf9d856e7904c1187f1803ef0f9ae93, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:41:15.143581 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e39894de-7043-4095-87e8-e6a3ec0a462f None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:41:15.156006 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 570/1153] 10.222.0.27 () {58 vars in 1439 bytes} [Sun Sep 11 00:41:15 2022] PUT /identity/v3/projects/9a9b03da26eb42aa9c9816873d8afb74/users/bdf9d856e7904c1187f1803ef0f9ae93/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:41:17.276002 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11f940af-5dd2-40c7-8fbf-bd65e8c891ee None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:17.276555 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11f940af-5dd2-40c7-8fbf-bd65e8c891ee None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:17.276555 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11f940af-5dd2-40c7-8fbf-bd65e8c891ee None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:17.277886 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 584/1154] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:41:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:41:17.283158 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c3be87-bec0-4e61-bc7a-4fb4fc950a6c None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:17.283588 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c3be87-bec0-4e61-bc7a-4fb4fc950a6c None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:17.283707 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c3be87-bec0-4e61-bc7a-4fb4fc950a6c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:17.302867 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-57c3be87-bec0-4e61-bc7a-4fb4fc950a6c None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:41:17.304036 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-57c3be87-bec0-4e61-bc7a-4fb4fc950a6c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:41:17.368346 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 571/1155] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:41:17 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:41:17.643051 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11f940af-5dd2-40c7-8fbf-bd65e8c891ee None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:41:17.743750 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11f940af-5dd2-40c7-8fbf-bd65e8c891ee None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:41:17.776231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d442fab-bd04-4c03-a054-a5b3c0c67f7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:41:17.776845 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d442fab-bd04-4c03-a054-a5b3c0c67f7d None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:17.777121 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d442fab-bd04-4c03-a054-a5b3c0c67f7d None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:17.777390 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d442fab-bd04-4c03-a054-a5b3c0c67f7d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:17.788313 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d442fab-bd04-4c03-a054-a5b3c0c67f7d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:41:17.793101 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-1d442fab-bd04-4c03-a054-a5b3c0c67f7d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:41:17.793101 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:41:17.793101 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:41:17.793101 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:41:17.793101 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:41:17.793101 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:41:17.793101 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:41:17.793101 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:41:17.793101 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:41:17.793101 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:41:17.793101 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:41:17.793101 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:41:17.793101 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:41:17.793101 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:41:17.793101 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:41:17.793101 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:41:17.794262 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:41:17.794262 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:41:17.794262 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:41:17.794262 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:41:17.794262 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:41:17.794262 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:41:17.794262 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:41:17.794262 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:41:17.794262 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:41:17.794262 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:41:17.794262 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:41:17.794262 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:41:17.794262 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:41:17.794262 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:41:17.794262 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:41:17.794262 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:41:17.795456 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:41:17.795456 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:41:17.795456 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:41:17.795456 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:41:17.795456 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:41:17.795456 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:41:17.795456 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:41:17.795456 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:41:17.795456 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:41:17.795456 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:41:17.795456 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:41:17.795456 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:41:17.798998 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 585/1156] 10.222.0.27 () {56 vars in 1096 bytes} [Sun Sep 11 00:41:17 2022] GET /identity/v3/roles/admin => generated 83 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:41:17.800329 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57c3be87-bec0-4e61-bc7a-4fb4fc950a6c None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:41:17.821753 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:41:17.899008 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57c3be87-bec0-4e61-bc7a-4fb4fc950a6c None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:41:17.932039 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74b95ea-6dbc-4727-9166-d3c565aeb229 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:41:17.932630 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74b95ea-6dbc-4727-9166-d3c565aeb229 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:17.932902 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74b95ea-6dbc-4727-9166-d3c565aeb229 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:17.933166 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74b95ea-6dbc-4727-9166-d3c565aeb229 None admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:17.933813 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f74b95ea-6dbc-4727-9166-d3c565aeb229 None admin] RBAC: Adding query filter params (name=admin) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:41:17.934107 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f74b95ea-6dbc-4727-9166-d3c565aeb229 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:41:17.935480 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f74b95ea-6dbc-4727-9166-d3c565aeb229 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:41:17.944184 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 572/1157] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:41:17 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:41:17.950010 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d442fab-bd04-4c03-a054-a5b3c0c67f7d None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:41:17.953693 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:41:18.047219 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d442fab-bd04-4c03-a054-a5b3c0c67f7d None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:41:18.079727 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-640eaad9-7984-47c5-a6d7-6a911abe03b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:41:18.080358 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-640eaad9-7984-47c5-a6d7-6a911abe03b3 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:18.080634 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-640eaad9-7984-47c5-a6d7-6a911abe03b3 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:18.080906 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-640eaad9-7984-47c5-a6d7-6a911abe03b3 None admin] PATH_INFO: `/v3/users/bdf9d856e7904c1187f1803ef0f9ae93` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:18.082958 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-640eaad9-7984-47c5-a6d7-6a911abe03b3 None admin] RBAC: Authorizing `identity:get_user(user_id=bdf9d856e7904c1187f1803ef0f9ae93)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:41:18.084387 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-640eaad9-7984-47c5-a6d7-6a911abe03b3 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:41:18.085977 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 586/1158] 10.222.0.27 () {56 vars in 1177 bytes} [Sun Sep 11 00:41:17 2022] GET /identity/v3/users/bdf9d856e7904c1187f1803ef0f9ae93 => generated 286 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:41:18.091794 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74b95ea-6dbc-4727-9166-d3c565aeb229 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:41:18.171793 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:41:18.189731 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74b95ea-6dbc-4727-9166-d3c565aeb229 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:41:18.222809 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe00826-0014-4f46-bcda-f4f819ebdfe3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:41:18.223408 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe00826-0014-4f46-bcda-f4f819ebdfe3 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:18.223684 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe00826-0014-4f46-bcda-f4f819ebdfe3 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:18.223953 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe00826-0014-4f46-bcda-f4f819ebdfe3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:18.288922 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbe00826-0014-4f46-bcda-f4f819ebdfe3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:41:18.290360 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbe00826-0014-4f46-bcda-f4f819ebdfe3 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:41:18.367444 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:41:18.386214 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 573/1159] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:41:18 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 295 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 11 00:41:18.392118 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-640eaad9-7984-47c5-a6d7-6a911abe03b3 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:41:18.460431 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:41:18.489360 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-640eaad9-7984-47c5-a6d7-6a911abe03b3 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:41:18.521464 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07e22e3-75c2-40e2-9e5c-288a1c1c9e3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:41:18.522072 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07e22e3-75c2-40e2-9e5c-288a1c1c9e3f None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:18.522348 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07e22e3-75c2-40e2-9e5c-288a1c1c9e3f None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:18.522616 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07e22e3-75c2-40e2-9e5c-288a1c1c9e3f None admin] PATH_INFO: `/v3/projects/9a9b03da26eb42aa9c9816873d8afb74` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:18.523743 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d07e22e3-75c2-40e2-9e5c-288a1c1c9e3f None admin] RBAC: Authorizing `identity:get_project(project_id=9a9b03da26eb42aa9c9816873d8afb74)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:41:18.525164 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d07e22e3-75c2-40e2-9e5c-288a1c1c9e3f None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:41:18.526768 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 587/1160] 10.222.0.27 () {56 vars in 1186 bytes} [Sun Sep 11 00:41:18 2022] GET /identity/v3/projects/9a9b03da26eb42aa9c9816873d8afb74 => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:41:18.532900 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe00826-0014-4f46-bcda-f4f819ebdfe3 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:41:18.629925 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe00826-0014-4f46-bcda-f4f819ebdfe3 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:41:18.663273 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba135f4-8612-49fb-a3c9-a5e6beb00fd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:41:18.663858 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba135f4-8612-49fb-a3c9-a5e6beb00fd2 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:18.664133 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba135f4-8612-49fb-a3c9-a5e6beb00fd2 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:18.664399 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba135f4-8612-49fb-a3c9-a5e6beb00fd2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:18.665115 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba135f4-8612-49fb-a3c9-a5e6beb00fd2 None admin] RBAC: Adding query filter params (role.id=7d9213c1187c4b5182bf8132cf23ea19, scope.project.id=9a9b03da26eb42aa9c9816873d8afb74, user.id=bdf9d856e7904c1187f1803ef0f9ae93) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:41:18.665417 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba135f4-8612-49fb-a3c9-a5e6beb00fd2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:41:18.666866 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba135f4-8612-49fb-a3c9-a5e6beb00fd2 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:41:18.672500 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 574/1161] 10.222.0.27 () {56 vars in 1374 bytes} [Sun Sep 11 00:41:18 2022] GET /identity/v3/role_assignments?user.id=bdf9d856e7904c1187f1803ef0f9ae93&scope.project.id=9a9b03da26eb42aa9c9816873d8afb74&role.id=7d9213c1187c4b5182bf8132cf23ea19 => generated 608 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:41:20.879153 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fe5f3d-d335-43df-a27e-e7f52b40c209 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:20.879719 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fe5f3d-d335-43df-a27e-e7f52b40c209 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:20.879719 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fe5f3d-d335-43df-a27e-e7f52b40c209 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:20.881053 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 588/1162] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:41:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:41:20.886162 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138a24e4-8cf4-47c8-a904-3ab545bd7587 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:20.886583 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138a24e4-8cf4-47c8-a904-3ab545bd7587 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:20.886684 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138a24e4-8cf4-47c8-a904-3ab545bd7587 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:20.904455 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-138a24e4-8cf4-47c8-a904-3ab545bd7587 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:41:20.905622 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-138a24e4-8cf4-47c8-a904-3ab545bd7587 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:41:20.947155 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:41:20.971128 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 575/1163] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:41:20 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:41:21.241461 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63fe5f3d-d335-43df-a27e-e7f52b40c209 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:41:21.340070 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63fe5f3d-d335-43df-a27e-e7f52b40c209 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:41:21.372274 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afcd5d68-c280-445d-b02c-74778dc7fe72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:41:21.372864 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afcd5d68-c280-445d-b02c-74778dc7fe72 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:21.373145 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afcd5d68-c280-445d-b02c-74778dc7fe72 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:21.373417 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afcd5d68-c280-445d-b02c-74778dc7fe72 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:21.384025 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:41:21.439330 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afcd5d68-c280-445d-b02c-74778dc7fe72 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:41:21.441512 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afcd5d68-c280-445d-b02c-74778dc7fe72 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:41:21.535926 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 589/1164] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:41:21 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 295 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 11 00:41:21.541874 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-138a24e4-8cf4-47c8-a904-3ab545bd7587 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:41:21.637002 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-138a24e4-8cf4-47c8-a904-3ab545bd7587 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:41:21.670165 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be98d650-db1f-42e7-8819-dfdef0b3f666 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:41:21.670778 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be98d650-db1f-42e7-8819-dfdef0b3f666 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:21.671081 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be98d650-db1f-42e7-8819-dfdef0b3f666 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:21.671352 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be98d650-db1f-42e7-8819-dfdef0b3f666 None admin] PATH_INFO: `/v3/domains/swift_test` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:21.678428 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be98d650-db1f-42e7-8819-dfdef0b3f666 None admin] RBAC: Authorizing `identity:get_domain(domain_id=swift_test)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:41:21.679916 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be98d650-db1f-42e7-8819-dfdef0b3f666 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:41:21.691184 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-be98d650-db1f-42e7-8819-dfdef0b3f666 None admin] Could not find domain: swift_test.: keystone.exception.DomainNotFound: Could not find domain: swift_test. Sep 11 00:41:21.691184 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:41:21.691184 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 712, in get_domain Sep 11 00:41:21.691184 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application project = self.driver.get_project(domain_id) Sep 11 00:41:21.691184 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 11 00:41:21.691184 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 11 00:41:21.691184 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 11 00:41:21.691184 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 11 00:41:21.691184 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: swift_test. Sep 11 00:41:21.691184 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:41:21.691184 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 11 00:41:21.691184 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:41:21.691184 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:41:21.691184 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:41:21.691184 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:41:21.691184 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:41:21.691184 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:41:21.692401 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:41:21.692401 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:41:21.692401 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:41:21.692401 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:41:21.692401 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:41:21.692401 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:41:21.692401 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:41:21.692401 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:41:21.692401 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:41:21.692401 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:41:21.692401 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:41:21.692401 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:41:21.692401 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:41:21.692401 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:41:21.692401 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:41:21.692401 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:41:21.693467 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:41:21.693467 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:41:21.693467 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:41:21.693467 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:41:21.693467 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:41:21.693467 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:41:21.693467 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:41:21.693467 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:41:21.693467 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:41:21.693467 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:41:21.693467 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 718, in get_domain Sep 11 00:41:21.693467 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:41:21.693467 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: swift_test. Sep 11 00:41:21.693467 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:41:21.693467 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 576/1165] 10.222.0.27 () {56 vars in 1117 bytes} [Sun Sep 11 00:41:21 2022] GET /identity/v3/domains/swift_test => generated 90 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:41:21.698246 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afcd5d68-c280-445d-b02c-74778dc7fe72 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:41:21.738775 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:41:21.793917 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afcd5d68-c280-445d-b02c-74778dc7fe72 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:41:21.826597 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de2bb86f-1905-4379-a87b-2f1f6340cf8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:41:21.827209 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de2bb86f-1905-4379-a87b-2f1f6340cf8a None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:21.827489 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de2bb86f-1905-4379-a87b-2f1f6340cf8a None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:21.827757 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de2bb86f-1905-4379-a87b-2f1f6340cf8a None admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:21.828387 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de2bb86f-1905-4379-a87b-2f1f6340cf8a None admin] RBAC: Adding query filter params (name=swift_test) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:41:21.828685 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de2bb86f-1905-4379-a87b-2f1f6340cf8a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:41:21.830031 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de2bb86f-1905-4379-a87b-2f1f6340cf8a None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:41:21.837566 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 590/1166] 10.222.0.27 () {56 vars in 1115 bytes} [Sun Sep 11 00:41:21 2022] GET /identity/v3/domains?name=swift_test => generated 125 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:41:23.985746 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c7542f-3600-4025-9394-1cb88a665d1c None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:23.986313 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c7542f-3600-4025-9394-1cb88a665d1c None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:23.986313 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c7542f-3600-4025-9394-1cb88a665d1c None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:23.988461 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 577/1167] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:41:23 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:41:23.993742 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b7f53a-11f4-4a31-b546-9bc7efbdb33f None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:23.994180 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b7f53a-11f4-4a31-b546-9bc7efbdb33f None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:23.994291 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b7f53a-11f4-4a31-b546-9bc7efbdb33f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:24.011908 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-69b7f53a-11f4-4a31-b546-9bc7efbdb33f None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:41:24.013115 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-69b7f53a-11f4-4a31-b546-9bc7efbdb33f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:41:24.075938 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 591/1168] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:41:23 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:41:24.348698 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c7542f-3600-4025-9394-1cb88a665d1c None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:41:24.384159 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:41:24.447883 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c7542f-3600-4025-9394-1cb88a665d1c None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:41:24.480863 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1f73fd-f6c0-429d-b705-3dfb8918b405 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:41:24.481445 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1f73fd-f6c0-429d-b705-3dfb8918b405 None admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:24.481717 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1f73fd-f6c0-429d-b705-3dfb8918b405 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:24.481979 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1f73fd-f6c0-429d-b705-3dfb8918b405 None admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:24.482629 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa1f73fd-f6c0-429d-b705-3dfb8918b405 None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:41:24.484008 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa1f73fd-f6c0-429d-b705-3dfb8918b405 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:41:24.499891 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 578/1169] 10.222.0.27 () {60 vars in 1138 bytes} [Sun Sep 11 00:41:24 2022] POST /identity/v3/domains => generated 267 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:41:26.703747 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58bdddb4-5e89-4e4e-a42a-988b92934a08 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:26.704320 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58bdddb4-5e89-4e4e-a42a-988b92934a08 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:26.704320 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58bdddb4-5e89-4e4e-a42a-988b92934a08 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:26.705687 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 592/1170] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:41:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:41:26.712610 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9b31ad-e901-452c-896c-ac4c04b07327 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:26.713072 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9b31ad-e901-452c-896c-ac4c04b07327 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:26.713174 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9b31ad-e901-452c-896c-ac4c04b07327 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:26.732504 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-ad9b31ad-e901-452c-896c-ac4c04b07327 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:41:26.734225 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-ad9b31ad-e901-452c-896c-ac4c04b07327 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:41:26.813496 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 579/1171] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:41:26 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:41:27.121494 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58bdddb4-5e89-4e4e-a42a-988b92934a08 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:41:27.157506 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:41:27.223538 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58bdddb4-5e89-4e4e-a42a-988b92934a08 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:41:27.255900 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9818c03-7e98-4f95-86f3-f40bbcc1b589 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:41:27.256517 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9818c03-7e98-4f95-86f3-f40bbcc1b589 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:27.256795 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9818c03-7e98-4f95-86f3-f40bbcc1b589 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:27.257065 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9818c03-7e98-4f95-86f3-f40bbcc1b589 None admin] PATH_INFO: `/v3/domains/bb010f669d7f44e6b612fbf7cfb7152c` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:27.267155 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9818c03-7e98-4f95-86f3-f40bbcc1b589 None admin] RBAC: Authorizing `identity:get_domain(domain_id=bb010f669d7f44e6b612fbf7cfb7152c)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:41:27.269415 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9818c03-7e98-4f95-86f3-f40bbcc1b589 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:41:27.271028 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 593/1172] 10.222.0.27 () {56 vars in 1183 bytes} [Sun Sep 11 00:41:27 2022] GET /identity/v3/domains/bb010f669d7f44e6b612fbf7cfb7152c => generated 267 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:41:27.276902 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9b31ad-e901-452c-896c-ac4c04b07327 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:41:27.304080 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:41:27.375440 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9b31ad-e901-452c-896c-ac4c04b07327 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:41:27.409339 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10458ef-9a57-4fa9-b389-e529d945f097 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:41:27.410197 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10458ef-9a57-4fa9-b389-e529d945f097 None admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:27.410465 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10458ef-9a57-4fa9-b389-e529d945f097 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:27.410727 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10458ef-9a57-4fa9-b389-e529d945f097 None admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:27.411394 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a10458ef-9a57-4fa9-b389-e529d945f097 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:41:27.412815 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a10458ef-9a57-4fa9-b389-e529d945f097 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:41:27.438073 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 580/1173] 10.222.0.27 () {60 vars in 1141 bytes} [Sun Sep 11 00:41:27 2022] POST /identity/v3/projects => generated 361 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:41:29.615437 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f617904-c980-42e2-84f4-ce6b79f72780 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:29.615979 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f617904-c980-42e2-84f4-ce6b79f72780 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:29.615979 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f617904-c980-42e2-84f4-ce6b79f72780 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:29.617332 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 594/1174] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:41:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:41:29.622482 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff17ba9a-c865-4b9b-9e89-b529c5817154 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:29.622953 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff17ba9a-c865-4b9b-9e89-b529c5817154 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:29.623049 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff17ba9a-c865-4b9b-9e89-b529c5817154 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:29.640961 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-ff17ba9a-c865-4b9b-9e89-b529c5817154 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:41:29.642119 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-ff17ba9a-c865-4b9b-9e89-b529c5817154 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:41:29.708414 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 581/1175] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:41:29 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:41:29.986092 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f617904-c980-42e2-84f4-ce6b79f72780 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:41:30.078088 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:41:30.086991 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f617904-c980-42e2-84f4-ce6b79f72780 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:41:30.120366 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce16009d-eeeb-4f24-bb3a-d1b6acfc6586 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:41:30.120991 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce16009d-eeeb-4f24-bb3a-d1b6acfc6586 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:30.121267 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce16009d-eeeb-4f24-bb3a-d1b6acfc6586 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:30.121536 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce16009d-eeeb-4f24-bb3a-d1b6acfc6586 None admin] PATH_INFO: `/v3/domains/bb010f669d7f44e6b612fbf7cfb7152c` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:30.122641 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce16009d-eeeb-4f24-bb3a-d1b6acfc6586 None admin] RBAC: Authorizing `identity:get_domain(domain_id=bb010f669d7f44e6b612fbf7cfb7152c)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:41:30.124099 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce16009d-eeeb-4f24-bb3a-d1b6acfc6586 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:41:30.125673 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 595/1176] 10.222.0.27 () {56 vars in 1183 bytes} [Sun Sep 11 00:41:29 2022] GET /identity/v3/domains/bb010f669d7f44e6b612fbf7cfb7152c => generated 267 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:41:30.131534 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff17ba9a-c865-4b9b-9e89-b529c5817154 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:41:30.141447 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:41:30.229169 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff17ba9a-c865-4b9b-9e89-b529c5817154 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:41:30.263407 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3692bf78-cb7e-4227-a084-d5609ab5cbc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:41:30.263931 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3692bf78-cb7e-4227-a084-d5609ab5cbc1 None admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:30.264203 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3692bf78-cb7e-4227-a084-d5609ab5cbc1 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:30.264470 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3692bf78-cb7e-4227-a084-d5609ab5cbc1 None admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:30.265117 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3692bf78-cb7e-4227-a084-d5609ab5cbc1 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:41:30.266531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3692bf78-cb7e-4227-a084-d5609ab5cbc1 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:41:30.289528 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 582/1177] 10.222.0.27 () {60 vars in 1132 bytes} [Sun Sep 11 00:41:30 2022] POST /identity/v3/users => generated 311 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:41:32.485757 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a2b55a-741f-4aa9-8475-60a0e2e7f659 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:32.486329 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a2b55a-741f-4aa9-8475-60a0e2e7f659 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:32.486329 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a2b55a-741f-4aa9-8475-60a0e2e7f659 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:32.488100 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 596/1178] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:41:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:41:32.493264 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cabf6df6-0f0f-43ac-bb40-bfe7a9b4a96e None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:32.493668 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cabf6df6-0f0f-43ac-bb40-bfe7a9b4a96e None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:32.493809 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cabf6df6-0f0f-43ac-bb40-bfe7a9b4a96e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:32.511551 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-cabf6df6-0f0f-43ac-bb40-bfe7a9b4a96e None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:41:32.512707 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-cabf6df6-0f0f-43ac-bb40-bfe7a9b4a96e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:41:32.570406 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:41:32.579138 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 583/1179] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:41:32 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:41:32.851613 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a2b55a-741f-4aa9-8475-60a0e2e7f659 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:41:32.950444 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a2b55a-741f-4aa9-8475-60a0e2e7f659 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:41:32.982843 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2beb51c-6f09-44eb-afc8-1aecf3fbe593 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:41:32.983461 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2beb51c-6f09-44eb-afc8-1aecf3fbe593 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:32.983743 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2beb51c-6f09-44eb-afc8-1aecf3fbe593 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:32.984018 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2beb51c-6f09-44eb-afc8-1aecf3fbe593 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:32.994804 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2beb51c-6f09-44eb-afc8-1aecf3fbe593 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:41:32.999646 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-b2beb51c-6f09-44eb-afc8-1aecf3fbe593 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:41:32.999646 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:41:32.999646 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:41:32.999646 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:41:32.999646 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:41:32.999646 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:41:32.999646 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:41:32.999646 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:41:32.999646 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:41:32.999646 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:41:32.999646 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:41:32.999646 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:41:32.999646 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:41:32.999646 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:41:32.999646 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:41:32.999646 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:41:33.000773 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:41:33.000773 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:41:33.000773 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:41:33.000773 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:41:33.000773 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:41:33.000773 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:41:33.000773 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:41:33.000773 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:41:33.000773 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:41:33.000773 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:41:33.000773 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:41:33.000773 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:41:33.000773 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:41:33.000773 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:41:33.000773 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:41:33.000773 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:41:33.001901 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:41:33.001901 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:41:33.001901 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:41:33.001901 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:41:33.001901 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:41:33.001901 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:41:33.001901 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:41:33.001901 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:41:33.001901 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:41:33.001901 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:41:33.001901 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:41:33.001901 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:41:33.001901 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 597/1180] 10.222.0.27 () {56 vars in 1096 bytes} [Sun Sep 11 00:41:32 2022] GET /identity/v3/roles/admin => generated 83 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:41:33.007179 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cabf6df6-0f0f-43ac-bb40-bfe7a9b4a96e None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:41:33.103949 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cabf6df6-0f0f-43ac-bb40-bfe7a9b4a96e None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:41:33.137569 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7427a0f-f6bf-41db-9a05-7da5f50c3ad7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:41:33.138161 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7427a0f-f6bf-41db-9a05-7da5f50c3ad7 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:33.138434 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7427a0f-f6bf-41db-9a05-7da5f50c3ad7 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:33.138701 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7427a0f-f6bf-41db-9a05-7da5f50c3ad7 None admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:33.139390 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7427a0f-f6bf-41db-9a05-7da5f50c3ad7 None admin] RBAC: Adding query filter params (name=admin) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:41:33.139688 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7427a0f-f6bf-41db-9a05-7da5f50c3ad7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:41:33.141040 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7427a0f-f6bf-41db-9a05-7da5f50c3ad7 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:41:33.148944 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 584/1181] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:41:33 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:41:33.155106 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2beb51c-6f09-44eb-afc8-1aecf3fbe593 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:41:33.179777 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:41:33.253013 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2beb51c-6f09-44eb-afc8-1aecf3fbe593 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:41:33.285416 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d711ca-3975-4f9e-9187-892903401ee1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:41:33.286023 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d711ca-3975-4f9e-9187-892903401ee1 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:33.286299 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d711ca-3975-4f9e-9187-892903401ee1 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:33.286564 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d711ca-3975-4f9e-9187-892903401ee1 None admin] PATH_INFO: `/v3/users/c0a4921135ef4047ae1fa923ba07ddad` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:33.298962 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94d711ca-3975-4f9e-9187-892903401ee1 None admin] RBAC: Authorizing `identity:get_user(user_id=c0a4921135ef4047ae1fa923ba07ddad)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:41:33.301253 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94d711ca-3975-4f9e-9187-892903401ee1 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:41:33.302830 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 598/1182] 10.222.0.27 () {56 vars in 1177 bytes} [Sun Sep 11 00:41:33 2022] GET /identity/v3/users/c0a4921135ef4047ae1fa923ba07ddad => generated 311 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:41:33.308541 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7427a0f-f6bf-41db-9a05-7da5f50c3ad7 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:41:33.363086 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:41:33.406058 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7427a0f-f6bf-41db-9a05-7da5f50c3ad7 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:41:33.438946 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db510f6f-eb4b-44ca-8965-746df971a334 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:41:33.439538 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db510f6f-eb4b-44ca-8965-746df971a334 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:33.439981 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db510f6f-eb4b-44ca-8965-746df971a334 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:33.440224 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db510f6f-eb4b-44ca-8965-746df971a334 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:33.507345 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db510f6f-eb4b-44ca-8965-746df971a334 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:41:33.508779 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db510f6f-eb4b-44ca-8965-746df971a334 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:41:33.561305 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:41:33.604300 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 585/1183] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:41:33 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 296 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 11 00:41:33.610321 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d711ca-3975-4f9e-9187-892903401ee1 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:41:33.688396 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:41:33.707221 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d711ca-3975-4f9e-9187-892903401ee1 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:41:33.740481 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abfe7547-212e-4d5e-bc4b-cff2bbd45b3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:41:33.741099 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abfe7547-212e-4d5e-bc4b-cff2bbd45b3f None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:33.741378 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abfe7547-212e-4d5e-bc4b-cff2bbd45b3f None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:33.741649 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abfe7547-212e-4d5e-bc4b-cff2bbd45b3f None admin] PATH_INFO: `/v3/projects/9b5a1e77666b4eb7be4b1a94ed062a9b` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:33.742750 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abfe7547-212e-4d5e-bc4b-cff2bbd45b3f None admin] RBAC: Authorizing `identity:get_project(project_id=9b5a1e77666b4eb7be4b1a94ed062a9b)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:41:33.744204 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abfe7547-212e-4d5e-bc4b-cff2bbd45b3f None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:41:33.745810 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 599/1184] 10.222.0.27 () {56 vars in 1186 bytes} [Sun Sep 11 00:41:33 2022] GET /identity/v3/projects/9b5a1e77666b4eb7be4b1a94ed062a9b => generated 361 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:41:33.751838 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db510f6f-eb4b-44ca-8965-746df971a334 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:41:33.847984 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db510f6f-eb4b-44ca-8965-746df971a334 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:41:33.879940 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e883de5-5f97-41ca-a4f8-f1ad711f70f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:41:33.880526 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e883de5-5f97-41ca-a4f8-f1ad711f70f1 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:33.880797 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e883de5-5f97-41ca-a4f8-f1ad711f70f1 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:33.881112 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e883de5-5f97-41ca-a4f8-f1ad711f70f1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:33.881826 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e883de5-5f97-41ca-a4f8-f1ad711f70f1 None admin] RBAC: Adding query filter params (role.id=7d9213c1187c4b5182bf8132cf23ea19, scope.project.id=9b5a1e77666b4eb7be4b1a94ed062a9b, user.id=c0a4921135ef4047ae1fa923ba07ddad) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:41:33.882126 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e883de5-5f97-41ca-a4f8-f1ad711f70f1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:41:33.883560 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e883de5-5f97-41ca-a4f8-f1ad711f70f1 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:41:33.888675 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 586/1185] 10.222.0.27 () {56 vars in 1374 bytes} [Sun Sep 11 00:41:33 2022] GET /identity/v3/role_assignments?user.id=c0a4921135ef4047ae1fa923ba07ddad&scope.project.id=9b5a1e77666b4eb7be4b1a94ed062a9b&role.id=7d9213c1187c4b5182bf8132cf23ea19 => generated 259 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:41:36.007586 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bacabc-d6b9-40d4-9d31-2216653317d5 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:36.008122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bacabc-d6b9-40d4-9d31-2216653317d5 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:36.008122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bacabc-d6b9-40d4-9d31-2216653317d5 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:36.009437 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 600/1186] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:41:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:41:36.014495 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8bc05d-fb5f-41bb-94cd-6bc03d5fa781 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:36.014954 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8bc05d-fb5f-41bb-94cd-6bc03d5fa781 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:36.015055 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8bc05d-fb5f-41bb-94cd-6bc03d5fa781 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:36.034936 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-9f8bc05d-fb5f-41bb-94cd-6bc03d5fa781 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:41:36.036100 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-9f8bc05d-fb5f-41bb-94cd-6bc03d5fa781 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:41:36.050578 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:41:36.102846 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 587/1187] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:41:36 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:41:36.368909 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1bacabc-d6b9-40d4-9d31-2216653317d5 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:41:36.466808 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1bacabc-d6b9-40d4-9d31-2216653317d5 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:41:36.498537 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e633e3ac-ffe5-4f16-b6f2-73fd191e06df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:41:36.499169 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e633e3ac-ffe5-4f16-b6f2-73fd191e06df None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:36.499443 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e633e3ac-ffe5-4f16-b6f2-73fd191e06df None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:36.499709 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e633e3ac-ffe5-4f16-b6f2-73fd191e06df None admin] PATH_INFO: `/v3/roles/admin` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:36.509907 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e633e3ac-ffe5-4f16-b6f2-73fd191e06df None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:41:36.514635 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-e633e3ac-ffe5-4f16-b6f2-73fd191e06df None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:41:36.514635 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:41:36.514635 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:41:36.514635 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:41:36.514635 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:41:36.514635 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:41:36.514635 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:41:36.514635 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:41:36.514635 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:41:36.514635 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:41:36.514635 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:41:36.514635 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:41:36.514635 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:41:36.514635 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:41:36.514635 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:41:36.514635 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:41:36.515756 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:41:36.515756 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:41:36.515756 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:41:36.515756 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:41:36.515756 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:41:36.515756 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:41:36.515756 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:41:36.515756 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:41:36.515756 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:41:36.515756 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:41:36.515756 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:41:36.515756 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:41:36.515756 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:41:36.515756 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:41:36.515756 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:41:36.515756 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:41:36.516821 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:41:36.516821 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:41:36.516821 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:41:36.516821 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:41:36.516821 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:41:36.516821 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:41:36.516821 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:41:36.516821 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:41:36.516821 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:41:36.516821 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:41:36.516821 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:41:36.516821 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:41:36.516821 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 601/1188] 10.222.0.27 () {56 vars in 1096 bytes} [Sun Sep 11 00:41:36 2022] GET /identity/v3/roles/admin => generated 83 bytes in 147 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:41:36.522341 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8bc05d-fb5f-41bb-94cd-6bc03d5fa781 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:41:36.617702 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8bc05d-fb5f-41bb-94cd-6bc03d5fa781 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:41:36.651224 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21010e49-c9aa-438e-af09-b40586768cd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:41:36.651757 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21010e49-c9aa-438e-af09-b40586768cd6 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:36.652034 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21010e49-c9aa-438e-af09-b40586768cd6 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:36.652303 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21010e49-c9aa-438e-af09-b40586768cd6 None admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:36.652959 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21010e49-c9aa-438e-af09-b40586768cd6 None admin] RBAC: Adding query filter params (name=admin) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:41:36.653259 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21010e49-c9aa-438e-af09-b40586768cd6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:41:36.654618 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21010e49-c9aa-438e-af09-b40586768cd6 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:41:36.663029 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 588/1189] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:41:36 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:41:36.668768 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e633e3ac-ffe5-4f16-b6f2-73fd191e06df None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:41:36.679786 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:41:36.765845 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e633e3ac-ffe5-4f16-b6f2-73fd191e06df None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:41:36.799372 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d687668-cce9-4722-9e27-0c5b44332dd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:41:36.799994 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d687668-cce9-4722-9e27-0c5b44332dd8 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:36.800266 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d687668-cce9-4722-9e27-0c5b44332dd8 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:36.800536 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d687668-cce9-4722-9e27-0c5b44332dd8 None admin] PATH_INFO: `/v3/users/c0a4921135ef4047ae1fa923ba07ddad` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:36.801649 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d687668-cce9-4722-9e27-0c5b44332dd8 None admin] RBAC: Authorizing `identity:get_user(user_id=c0a4921135ef4047ae1fa923ba07ddad)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:41:36.803103 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d687668-cce9-4722-9e27-0c5b44332dd8 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:41:36.804666 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 602/1190] 10.222.0.27 () {56 vars in 1177 bytes} [Sun Sep 11 00:41:36 2022] GET /identity/v3/users/c0a4921135ef4047ae1fa923ba07ddad => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:41:36.810488 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21010e49-c9aa-438e-af09-b40586768cd6 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:41:36.826158 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:41:36.912227 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21010e49-c9aa-438e-af09-b40586768cd6 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:41:36.943918 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554d9635-ac13-4fe1-b917-cfb70c11e45f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:41:36.944530 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554d9635-ac13-4fe1-b917-cfb70c11e45f None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:36.944805 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554d9635-ac13-4fe1-b917-cfb70c11e45f None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:36.945073 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554d9635-ac13-4fe1-b917-cfb70c11e45f None admin] PATH_INFO: `/v3/projects/9b5a1e77666b4eb7be4b1a94ed062a9b` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:36.946174 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-554d9635-ac13-4fe1-b917-cfb70c11e45f None admin] RBAC: Authorizing `identity:get_project(project_id=9b5a1e77666b4eb7be4b1a94ed062a9b)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:41:36.947624 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-554d9635-ac13-4fe1-b917-cfb70c11e45f None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:41:36.949230 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 589/1191] 10.222.0.27 () {56 vars in 1186 bytes} [Sun Sep 11 00:41:36 2022] GET /identity/v3/projects/9b5a1e77666b4eb7be4b1a94ed062a9b => generated 361 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:41:36.955751 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d687668-cce9-4722-9e27-0c5b44332dd8 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:41:37.029510 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:41:37.052708 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d687668-cce9-4722-9e27-0c5b44332dd8 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:41:37.085546 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3808a1c9-3b48-4fb5-bb24-dbd1816ba6a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:41:37.086199 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3808a1c9-3b48-4fb5-bb24-dbd1816ba6a7 None admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:37.086474 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3808a1c9-3b48-4fb5-bb24-dbd1816ba6a7 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:37.086743 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3808a1c9-3b48-4fb5-bb24-dbd1816ba6a7 None admin] PATH_INFO: `/v3/projects/9b5a1e77666b4eb7be4b1a94ed062a9b/users/c0a4921135ef4047ae1fa923ba07ddad/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:37.088641 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3808a1c9-3b48-4fb5-bb24-dbd1816ba6a7 None admin] RBAC: Authorizing `identity:create_grant(project_id=9b5a1e77666b4eb7be4b1a94ed062a9b, user_id=c0a4921135ef4047ae1fa923ba07ddad, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:41:37.090058 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3808a1c9-3b48-4fb5-bb24-dbd1816ba6a7 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:41:37.102980 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 603/1192] 10.222.0.27 () {58 vars in 1439 bytes} [Sun Sep 11 00:41:36 2022] PUT /identity/v3/projects/9b5a1e77666b4eb7be4b1a94ed062a9b/users/c0a4921135ef4047ae1fa923ba07ddad/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:41:39.226946 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea2a565-e28b-4aa9-a08d-c71bb26b1260 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:39.227505 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea2a565-e28b-4aa9-a08d-c71bb26b1260 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:39.227505 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea2a565-e28b-4aa9-a08d-c71bb26b1260 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:39.228829 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 590/1193] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:41:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:41:39.233911 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1c20eb-2703-44b3-a79d-93a4bdfac6cb None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:39.234317 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1c20eb-2703-44b3-a79d-93a4bdfac6cb None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:39.234457 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1c20eb-2703-44b3-a79d-93a4bdfac6cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:39.253423 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-2e1c20eb-2703-44b3-a79d-93a4bdfac6cb None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:41:39.254594 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-2e1c20eb-2703-44b3-a79d-93a4bdfac6cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:41:39.320079 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 604/1194] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:41:39 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:41:39.591273 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea2a565-e28b-4aa9-a08d-c71bb26b1260 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:41:39.671523 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:41:39.691963 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea2a565-e28b-4aa9-a08d-c71bb26b1260 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:41:39.725130 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a050f73-6014-4d03-9fc1-fcab20f5ffbb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:41:39.725742 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a050f73-6014-4d03-9fc1-fcab20f5ffbb None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:39.726018 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a050f73-6014-4d03-9fc1-fcab20f5ffbb None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:39.726287 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a050f73-6014-4d03-9fc1-fcab20f5ffbb None admin] PATH_INFO: `/v3/roles/admin` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:39.737150 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a050f73-6014-4d03-9fc1-fcab20f5ffbb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:41:39.741965 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-1a050f73-6014-4d03-9fc1-fcab20f5ffbb None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:41:39.741965 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:41:39.741965 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:41:39.741965 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:41:39.741965 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:41:39.741965 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:41:39.741965 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:41:39.741965 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:41:39.741965 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:41:39.741965 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:41:39.741965 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:41:39.741965 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:41:39.741965 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:41:39.741965 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:41:39.741965 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:41:39.741965 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:41:39.746884 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:41:39.746884 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:41:39.746884 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:41:39.746884 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:41:39.746884 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:41:39.746884 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:41:39.746884 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:41:39.746884 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:41:39.746884 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:41:39.746884 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:41:39.746884 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:41:39.746884 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:41:39.746884 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:41:39.746884 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:41:39.746884 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:41:39.746884 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:41:39.747948 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:41:39.747948 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:41:39.747948 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:41:39.747948 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:41:39.747948 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:41:39.747948 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:41:39.747948 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:41:39.747948 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:41:39.747948 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:41:39.747948 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:41:39.747948 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:41:39.747948 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:41:39.747948 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 591/1195] 10.222.0.27 () {56 vars in 1096 bytes} [Sun Sep 11 00:41:39 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:41:39.749063 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1c20eb-2703-44b3-a79d-93a4bdfac6cb None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:41:39.810520 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:41:39.844416 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1c20eb-2703-44b3-a79d-93a4bdfac6cb None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:41:39.876855 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7047e75-7656-4028-99b1-d9a65059b80d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:41:39.877979 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7047e75-7656-4028-99b1-d9a65059b80d None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:39.878250 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7047e75-7656-4028-99b1-d9a65059b80d None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:39.878516 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7047e75-7656-4028-99b1-d9a65059b80d None admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:39.879197 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7047e75-7656-4028-99b1-d9a65059b80d None admin] RBAC: Adding query filter params (name=admin) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:41:39.879494 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7047e75-7656-4028-99b1-d9a65059b80d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:41:39.880839 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7047e75-7656-4028-99b1-d9a65059b80d None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:41:39.888474 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 605/1196] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:41:39 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:41:39.894308 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a050f73-6014-4d03-9fc1-fcab20f5ffbb None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:41:39.990400 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a050f73-6014-4d03-9fc1-fcab20f5ffbb None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:41:40.022831 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b5dbb7b-378e-4ac8-839f-466c936dda5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:41:40.023440 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b5dbb7b-378e-4ac8-839f-466c936dda5e None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:40.023716 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b5dbb7b-378e-4ac8-839f-466c936dda5e None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:40.023987 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b5dbb7b-378e-4ac8-839f-466c936dda5e None admin] PATH_INFO: `/v3/users/c0a4921135ef4047ae1fa923ba07ddad` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:40.025085 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b5dbb7b-378e-4ac8-839f-466c936dda5e None admin] RBAC: Authorizing `identity:get_user(user_id=c0a4921135ef4047ae1fa923ba07ddad)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:41:40.026504 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b5dbb7b-378e-4ac8-839f-466c936dda5e None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:41:40.028192 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 592/1197] 10.222.0.27 () {56 vars in 1177 bytes} [Sun Sep 11 00:41:39 2022] GET /identity/v3/users/c0a4921135ef4047ae1fa923ba07ddad => generated 311 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:41:40.033857 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7047e75-7656-4028-99b1-d9a65059b80d None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:41:40.131428 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7047e75-7656-4028-99b1-d9a65059b80d None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:41:40.155670 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:41:40.164210 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa28f0fe-1930-4121-b641-ad78da911c39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:41:40.164817 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa28f0fe-1930-4121-b641-ad78da911c39 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:40.165098 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa28f0fe-1930-4121-b641-ad78da911c39 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:40.165372 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa28f0fe-1930-4121-b641-ad78da911c39 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:40.229834 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa28f0fe-1930-4121-b641-ad78da911c39 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:41:40.231303 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa28f0fe-1930-4121-b641-ad78da911c39 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:41:40.325963 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 606/1198] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:41:40 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 11 00:41:40.332456 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b5dbb7b-378e-4ac8-839f-466c936dda5e None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:41:40.343663 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:41:40.431669 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b5dbb7b-378e-4ac8-839f-466c936dda5e None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:41:40.463950 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f75760-c9d3-4599-88c0-7e428edfa92e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:41:40.464556 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f75760-c9d3-4599-88c0-7e428edfa92e None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:40.464831 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f75760-c9d3-4599-88c0-7e428edfa92e None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:40.465097 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f75760-c9d3-4599-88c0-7e428edfa92e None admin] PATH_INFO: `/v3/projects/9b5a1e77666b4eb7be4b1a94ed062a9b` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:40.466195 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84f75760-c9d3-4599-88c0-7e428edfa92e None admin] RBAC: Authorizing `identity:get_project(project_id=9b5a1e77666b4eb7be4b1a94ed062a9b)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:41:40.467665 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84f75760-c9d3-4599-88c0-7e428edfa92e None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:41:40.469295 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 593/1199] 10.222.0.27 () {56 vars in 1186 bytes} [Sun Sep 11 00:41:40 2022] GET /identity/v3/projects/9b5a1e77666b4eb7be4b1a94ed062a9b => generated 361 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:41:40.476063 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa28f0fe-1930-4121-b641-ad78da911c39 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:41:40.503303 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:41:40.572489 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa28f0fe-1930-4121-b641-ad78da911c39 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:41:40.605930 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e80f6e-f123-4ae4-a1fc-4a7e802e32a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:41:40.606527 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e80f6e-f123-4ae4-a1fc-4a7e802e32a9 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:41:40.606828 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e80f6e-f123-4ae4-a1fc-4a7e802e32a9 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:41:40.607108 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e80f6e-f123-4ae4-a1fc-4a7e802e32a9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:41:40.607827 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40e80f6e-f123-4ae4-a1fc-4a7e802e32a9 None admin] RBAC: Adding query filter params (role.id=7d9213c1187c4b5182bf8132cf23ea19, scope.project.id=9b5a1e77666b4eb7be4b1a94ed062a9b, user.id=c0a4921135ef4047ae1fa923ba07ddad) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:41:40.608135 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40e80f6e-f123-4ae4-a1fc-4a7e802e32a9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:41:40.609549 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40e80f6e-f123-4ae4-a1fc-4a7e802e32a9 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:41:40.615197 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 607/1200] 10.222.0.27 () {56 vars in 1374 bytes} [Sun Sep 11 00:41:40 2022] GET /identity/v3/role_assignments?user.id=c0a4921135ef4047ae1fa923ba07ddad&scope.project.id=9b5a1e77666b4eb7be4b1a94ed062a9b&role.id=7d9213c1187c4b5182bf8132cf23ea19 => generated 608 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:41:50.459847 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=62 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:41:50.612675 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=54 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:42:36.497710 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7103aa54-143a-4ac7-a299-d7f2223b8c68 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:36.498344 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7103aa54-143a-4ac7-a299-d7f2223b8c68 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:36.498344 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7103aa54-143a-4ac7-a299-d7f2223b8c68 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:36.500257 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 594/1201] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:42:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:42:36.507031 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5586c0d2-4414-4802-ade0-aaffd40721be None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:36.507537 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5586c0d2-4414-4802-ade0-aaffd40721be None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:36.507766 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5586c0d2-4414-4802-ade0-aaffd40721be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:36.535520 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-5586c0d2-4414-4802-ade0-aaffd40721be None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:42:36.536711 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-5586c0d2-4414-4802-ade0-aaffd40721be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:42:36.624493 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 608/1202] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:42:36 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:42:36.950900 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7103aa54-143a-4ac7-a299-d7f2223b8c68 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:42:37.079999 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7103aa54-143a-4ac7-a299-d7f2223b8c68 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:42:37.119843 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc313666-7107-47cc-b6a9-2ce395ef22e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:42:37.120495 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc313666-7107-47cc-b6a9-2ce395ef22e7 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:37.120770 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc313666-7107-47cc-b6a9-2ce395ef22e7 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:37.121037 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc313666-7107-47cc-b6a9-2ce395ef22e7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:37.137099 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc313666-7107-47cc-b6a9-2ce395ef22e7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:42:37.138562 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc313666-7107-47cc-b6a9-2ce395ef22e7 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:42:37.160526 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-cc313666-7107-47cc-b6a9-2ce395ef22e7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:42:37.160526 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:42:37.160526 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:42:37.160526 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:42:37.160526 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:42:37.160526 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:42:37.160526 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:42:37.160526 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:42:37.160526 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:42:37.160526 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:42:37.160526 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:42:37.160526 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:42:37.160526 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:42:37.160526 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:42:37.160526 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:42:37.160526 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:42:37.163102 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:42:37.163102 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:42:37.163102 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:42:37.163102 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:42:37.163102 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:42:37.163102 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:42:37.163102 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:42:37.163102 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:42:37.163102 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:42:37.163102 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:42:37.163102 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:42:37.163102 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:42:37.163102 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:42:37.163102 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:42:37.163102 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:42:37.163102 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:42:37.163102 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:42:37.164151 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:42:37.164151 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:42:37.164151 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:42:37.164151 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 595/1203] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:42:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:42:37.168859 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5586c0d2-4414-4802-ade0-aaffd40721be None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:42:37.309636 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5586c0d2-4414-4802-ade0-aaffd40721be None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:42:37.346703 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:42:37.360570 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3af22a5-06d3-40b6-ae9d-3ce5c1f99512 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:42:37.361241 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3af22a5-06d3-40b6-ae9d-3ce5c1f99512 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:37.361629 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3af22a5-06d3-40b6-ae9d-3ce5c1f99512 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:37.361911 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3af22a5-06d3-40b6-ae9d-3ce5c1f99512 None admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:37.362587 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3af22a5-06d3-40b6-ae9d-3ce5c1f99512 None admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:42:37.362918 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3af22a5-06d3-40b6-ae9d-3ce5c1f99512 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:42:37.364301 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3af22a5-06d3-40b6-ae9d-3ce5c1f99512 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:42:37.384102 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 609/1204] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:42:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:42:37.385165 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc313666-7107-47cc-b6a9-2ce395ef22e7 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:42:37.457474 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:42:37.513317 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc313666-7107-47cc-b6a9-2ce395ef22e7 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:42:37.565187 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4751119-b409-44c7-b640-2563fbd70507 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:42:37.565945 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4751119-b409-44c7-b640-2563fbd70507 None admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:37.566392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4751119-b409-44c7-b640-2563fbd70507 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:37.566810 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4751119-b409-44c7-b640-2563fbd70507 None admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:37.567479 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4751119-b409-44c7-b640-2563fbd70507 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:42:37.569561 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4751119-b409-44c7-b640-2563fbd70507 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:42:37.598836 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 596/1205] 10.222.0.27 () {60 vars in 1132 bytes} [Sun Sep 11 00:42:37 2022] POST /identity/v3/users => generated 251 bytes in 213 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:42:39.999931 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79eede29-5648-4c2a-92ec-c8e1387e2bc9 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:40.000578 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79eede29-5648-4c2a-92ec-c8e1387e2bc9 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:40.000578 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79eede29-5648-4c2a-92ec-c8e1387e2bc9 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:40.001999 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 610/1206] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:42:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:42:40.007776 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42d33df-7b24-47f0-bc39-4194e9b534ac None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:40.008252 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42d33df-7b24-47f0-bc39-4194e9b534ac None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:40.008460 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42d33df-7b24-47f0-bc39-4194e9b534ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:40.029063 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-a42d33df-7b24-47f0-bc39-4194e9b534ac None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:42:40.030316 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-a42d33df-7b24-47f0-bc39-4194e9b534ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:42:40.118972 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 597/1207] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:42:40 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:42:40.392438 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79eede29-5648-4c2a-92ec-c8e1387e2bc9 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:42:40.510460 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79eede29-5648-4c2a-92ec-c8e1387e2bc9 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:42:40.556856 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80269dda-0ec2-4a1e-ba12-768c447f3591 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:42:40.557496 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80269dda-0ec2-4a1e-ba12-768c447f3591 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:40.557927 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80269dda-0ec2-4a1e-ba12-768c447f3591 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:40.558039 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80269dda-0ec2-4a1e-ba12-768c447f3591 None admin] PATH_INFO: `/v3/roles/service` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:40.569992 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80269dda-0ec2-4a1e-ba12-768c447f3591 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:42:40.576037 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-80269dda-0ec2-4a1e-ba12-768c447f3591 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 11 00:42:40.576037 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:42:40.576037 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:42:40.576037 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:42:40.576037 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:42:40.576037 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:42:40.576037 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:42:40.576037 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:42:40.576037 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:42:40.576037 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:42:40.576037 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:42:40.576037 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:42:40.576037 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:42:40.576037 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:42:40.576037 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:42:40.576037 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:42:40.577736 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:42:40.577736 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:42:40.577736 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:42:40.577736 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:42:40.577736 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:42:40.577736 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:42:40.577736 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:42:40.577736 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:42:40.577736 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:42:40.577736 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:42:40.577736 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:42:40.577736 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:42:40.577736 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:42:40.577736 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:42:40.577736 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:42:40.577736 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:42:40.578622 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:42:40.578622 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:42:40.578622 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:42:40.578622 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:42:40.578622 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:42:40.578622 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:42:40.578622 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:42:40.578622 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:42:40.578622 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:42:40.578622 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:42:40.578622 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 11 00:42:40.578622 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:42:40.578622 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 611/1208] 10.222.0.27 () {56 vars in 1102 bytes} [Sun Sep 11 00:42:40 2022] GET /identity/v3/roles/service => generated 85 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:42:40.583980 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a42d33df-7b24-47f0-bc39-4194e9b534ac None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:42:40.633482 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:42:40.704873 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a42d33df-7b24-47f0-bc39-4194e9b534ac None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:42:40.750663 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a91dbd-c308-4296-b1f1-ee331ed12451 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:42:40.751313 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a91dbd-c308-4296-b1f1-ee331ed12451 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:40.751592 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a91dbd-c308-4296-b1f1-ee331ed12451 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:40.751864 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a91dbd-c308-4296-b1f1-ee331ed12451 None admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:40.752540 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26a91dbd-c308-4296-b1f1-ee331ed12451 None admin] RBAC: Adding query filter params (name=service) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:42:40.752843 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26a91dbd-c308-4296-b1f1-ee331ed12451 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:42:40.754217 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26a91dbd-c308-4296-b1f1-ee331ed12451 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:42:40.763746 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 598/1209] 10.222.0.27 () {56 vars in 1103 bytes} [Sun Sep 11 00:42:40 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:42:40.770758 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80269dda-0ec2-4a1e-ba12-768c447f3591 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:42:40.829500 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:42:40.895044 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80269dda-0ec2-4a1e-ba12-768c447f3591 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:42:40.935060 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b707b8e-200d-42c8-94bb-ba0e3738a856 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:42:40.935840 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b707b8e-200d-42c8-94bb-ba0e3738a856 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:40.936263 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b707b8e-200d-42c8-94bb-ba0e3738a856 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:40.936751 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b707b8e-200d-42c8-94bb-ba0e3738a856 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:40.948512 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b707b8e-200d-42c8-94bb-ba0e3738a856 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:42:40.950150 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b707b8e-200d-42c8-94bb-ba0e3738a856 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:42:40.966458 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-6b707b8e-200d-42c8-94bb-ba0e3738a856 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:42:40.966458 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:42:40.966458 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:42:40.966458 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:42:40.966458 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:42:40.966458 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:42:40.966458 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:42:40.966458 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:42:40.966458 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:42:40.966458 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:42:40.966458 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:42:40.966458 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:42:40.966458 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:42:40.966458 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:42:40.966458 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:42:40.966458 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:42:40.968533 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:42:40.968533 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:42:40.968533 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:42:40.968533 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:42:40.968533 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:42:40.968533 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:42:40.968533 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:42:40.968533 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:42:40.968533 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:42:40.968533 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:42:40.968533 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:42:40.968533 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:42:40.968533 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:42:40.968533 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:42:40.968533 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:42:40.968533 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:42:40.968533 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:42:40.969928 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:42:40.969928 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:42:40.969928 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:42:40.975171 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 612/1210] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:42:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:42:40.977003 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a91dbd-c308-4296-b1f1-ee331ed12451 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:42:41.100917 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:42:41.105884 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a91dbd-c308-4296-b1f1-ee331ed12451 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:42:41.147722 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd796ae-779e-4a4e-9ac3-e709aa4079ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:42:41.148442 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd796ae-779e-4a4e-9ac3-e709aa4079ad None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:41.148782 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd796ae-779e-4a4e-9ac3-e709aa4079ad None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:41.149058 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd796ae-779e-4a4e-9ac3-e709aa4079ad None admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:41.149705 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfd796ae-779e-4a4e-9ac3-e709aa4079ad None admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:42:41.150010 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfd796ae-779e-4a4e-9ac3-e709aa4079ad None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:42:41.151419 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfd796ae-779e-4a4e-9ac3-e709aa4079ad None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:42:41.166827 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 599/1211] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:42:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:42:41.172953 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b707b8e-200d-42c8-94bb-ba0e3738a856 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:42:41.285872 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b707b8e-200d-42c8-94bb-ba0e3738a856 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:42:41.287433 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:42:41.324634 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f707dd47-3893-4e38-9bd5-693cab83360d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:42:41.326007 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f707dd47-3893-4e38-9bd5-693cab83360d None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:41.326007 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f707dd47-3893-4e38-9bd5-693cab83360d None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:41.326007 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f707dd47-3893-4e38-9bd5-693cab83360d None admin] PATH_INFO: `/v3/users/placement` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:41.334908 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f707dd47-3893-4e38-9bd5-693cab83360d None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:42:41.338690 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f707dd47-3893-4e38-9bd5-693cab83360d None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:42:41.350575 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-f707dd47-3893-4e38-9bd5-693cab83360d None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 11 00:42:41.350575 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:42:41.350575 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:42:41.350575 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:42:41.350575 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:42:41.350575 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:42:41.350575 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:42:41.350575 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:42:41.350575 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:42:41.350575 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:42:41.350575 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:42:41.350575 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:42:41.350575 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 11 00:42:41.350575 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 11 00:42:41.350575 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 11 00:42:41.350575 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 11 00:42:41.351871 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:42:41.351871 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:42:41.351871 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 11 00:42:41.351871 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:42:41.351871 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 11 00:42:41.351871 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:42:41.351871 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:42:41.351871 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:42:41.351871 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:42:41.351871 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:42:41.351871 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:42:41.351871 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:42:41.351871 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:42:41.351871 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:42:41.351871 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:42:41.351871 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:42:41.351871 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:42:41.351871 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:42:41.352890 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:42:41.352890 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:42:41.352890 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 11 00:42:41.352890 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 11 00:42:41.352890 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 11 00:42:41.352890 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 11 00:42:41.352890 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 11 00:42:41.352890 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 11 00:42:41.352890 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 11 00:42:41.352890 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:42:41.352890 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 613/1212] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:42:41 2022] GET /identity/v3/users/placement => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:42:41.357660 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd796ae-779e-4a4e-9ac3-e709aa4079ad None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:42:41.484151 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd796ae-779e-4a4e-9ac3-e709aa4079ad None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:42:41.525241 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-652d4b4e-1e56-43d1-b866-c53849cec41b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:42:41.526009 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652d4b4e-1e56-43d1-b866-c53849cec41b None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:41.526289 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652d4b4e-1e56-43d1-b866-c53849cec41b None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:41.526606 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652d4b4e-1e56-43d1-b866-c53849cec41b None admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:41.527431 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-652d4b4e-1e56-43d1-b866-c53849cec41b None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:42:41.527866 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-652d4b4e-1e56-43d1-b866-c53849cec41b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:42:41.529581 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-652d4b4e-1e56-43d1-b866-c53849cec41b None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:42:41.551735 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 600/1213] 10.222.0.27 () {56 vars in 1143 bytes} [Sun Sep 11 00:42:41 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:42:41.557516 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f707dd47-3893-4e38-9bd5-693cab83360d None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:42:41.670690 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f707dd47-3893-4e38-9bd5-693cab83360d None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:42:41.709697 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2454a901-6b46-4c73-8cfb-45cdfc301701 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:42:41.710421 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2454a901-6b46-4c73-8cfb-45cdfc301701 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:41.710889 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2454a901-6b46-4c73-8cfb-45cdfc301701 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:41.711162 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2454a901-6b46-4c73-8cfb-45cdfc301701 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:41.745556 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:42:41.799000 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2454a901-6b46-4c73-8cfb-45cdfc301701 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:42:41.800904 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2454a901-6b46-4c73-8cfb-45cdfc301701 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:42:41.923574 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 614/1214] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:42:41 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 366 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 11 00:42:41.929502 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-652d4b4e-1e56-43d1-b866-c53849cec41b None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:42:41.951543 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:42:42.047584 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-652d4b4e-1e56-43d1-b866-c53849cec41b None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:42:42.090041 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb322c8-a1a0-44ca-95a8-1a6b11d0ad06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:42:42.090743 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb322c8-a1a0-44ca-95a8-1a6b11d0ad06 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:42.091058 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb322c8-a1a0-44ca-95a8-1a6b11d0ad06 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:42.091331 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb322c8-a1a0-44ca-95a8-1a6b11d0ad06 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:42.105253 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eb322c8-a1a0-44ca-95a8-1a6b11d0ad06 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:42:42.106779 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eb322c8-a1a0-44ca-95a8-1a6b11d0ad06 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:42:42.123796 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-3eb322c8-a1a0-44ca-95a8-1a6b11d0ad06 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:42:42.123796 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:42:42.123796 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:42:42.123796 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:42:42.123796 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:42:42.123796 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:42:42.123796 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:42:42.123796 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:42:42.123796 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:42:42.123796 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:42:42.123796 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:42:42.123796 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:42:42.123796 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:42:42.123796 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:42:42.123796 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:42:42.123796 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:42:42.129647 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:42:42.129647 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:42:42.129647 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:42:42.129647 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:42:42.129647 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:42:42.129647 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:42:42.129647 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:42:42.129647 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:42:42.129647 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:42:42.129647 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:42:42.129647 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:42:42.129647 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:42:42.129647 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:42:42.129647 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:42:42.129647 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:42:42.129647 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:42:42.129647 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:42:42.130581 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:42:42.130581 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:42:42.130581 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:42:42.130581 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 601/1215] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:42:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:42:42.130861 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2454a901-6b46-4c73-8cfb-45cdfc301701 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:42:42.205014 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:42:42.252179 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2454a901-6b46-4c73-8cfb-45cdfc301701 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:42:42.294542 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5141819-3798-4e50-8343-512359464afe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:42:42.295200 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5141819-3798-4e50-8343-512359464afe None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:42.296060 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5141819-3798-4e50-8343-512359464afe None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:42.296502 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5141819-3798-4e50-8343-512359464afe None admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:42.297153 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5141819-3798-4e50-8343-512359464afe None admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:42:42.297465 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5141819-3798-4e50-8343-512359464afe None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:42:42.298988 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5141819-3798-4e50-8343-512359464afe None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:42:42.312258 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 615/1216] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:42:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:42:42.318354 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb322c8-a1a0-44ca-95a8-1a6b11d0ad06 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:42:42.403069 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:42:42.448026 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb322c8-a1a0-44ca-95a8-1a6b11d0ad06 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:42:42.488565 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e2e04b-d44a-4d33-9c43-0d8ff8b973c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:42:42.489178 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e2e04b-d44a-4d33-9c43-0d8ff8b973c5 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:42.489496 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e2e04b-d44a-4d33-9c43-0d8ff8b973c5 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:42.489807 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e2e04b-d44a-4d33-9c43-0d8ff8b973c5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:42.498138 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e2e04b-d44a-4d33-9c43-0d8ff8b973c5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:42:42.500936 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e2e04b-d44a-4d33-9c43-0d8ff8b973c5 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:42:42.515609 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-f1e2e04b-d44a-4d33-9c43-0d8ff8b973c5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:42:42.515609 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:42:42.515609 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:42:42.515609 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:42:42.515609 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:42:42.515609 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:42:42.515609 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:42:42.515609 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:42:42.515609 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:42:42.515609 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:42:42.515609 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:42:42.515609 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:42:42.515609 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 11 00:42:42.515609 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 11 00:42:42.515609 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 11 00:42:42.517200 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 11 00:42:42.517200 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:42:42.517200 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:42:42.517200 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:42:42.517200 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:42:42.517200 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:42:42.517200 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:42:42.517200 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:42:42.517200 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:42:42.517200 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:42:42.517200 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:42:42.517200 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:42:42.517200 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:42:42.517200 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:42:42.517200 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:42:42.517200 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:42:42.517200 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:42:42.518314 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 11 00:42:42.518314 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 11 00:42:42.518314 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 11 00:42:42.518314 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 11 00:42:42.518314 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 11 00:42:42.518314 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 11 00:42:42.518314 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:42:42.518314 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:42:42.518877 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 602/1217] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:42:42 2022] GET /identity/v3/projects/service => generated 88 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:42:42.524163 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5141819-3798-4e50-8343-512359464afe None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:42:42.651064 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5141819-3798-4e50-8343-512359464afe None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:42:42.684063 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:42:42.699606 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07cca91-b67f-424c-ad1b-f2b5270cd3b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:42:42.700471 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07cca91-b67f-424c-ad1b-f2b5270cd3b3 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:42.700902 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07cca91-b67f-424c-ad1b-f2b5270cd3b3 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:42.701173 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07cca91-b67f-424c-ad1b-f2b5270cd3b3 None admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:42.701906 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a07cca91-b67f-424c-ad1b-f2b5270cd3b3 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:42:42.702202 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a07cca91-b67f-424c-ad1b-f2b5270cd3b3 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:42:42.703738 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a07cca91-b67f-424c-ad1b-f2b5270cd3b3 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:42:42.718486 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 616/1218] 10.222.0.27 () {56 vars in 1148 bytes} [Sun Sep 11 00:42:42 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:42:42.725005 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e2e04b-d44a-4d33-9c43-0d8ff8b973c5 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:42:42.834587 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e2e04b-d44a-4d33-9c43-0d8ff8b973c5 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:42:42.867237 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:42:42.874759 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b35f04d-ba91-4cd1-81f9-9ef4d0966ff0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:42:42.875394 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b35f04d-ba91-4cd1-81f9-9ef4d0966ff0 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:42.875661 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b35f04d-ba91-4cd1-81f9-9ef4d0966ff0 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:42.875927 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b35f04d-ba91-4cd1-81f9-9ef4d0966ff0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:42.876643 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b35f04d-ba91-4cd1-81f9-9ef4d0966ff0 None admin] RBAC: Adding query filter params (role.id=0390f926e7ef4715ae18697033bde90e, scope.project.id=b504a5058bc6474c8cc077dfeec16799, user.id=cdbfc2ccd73e4a5eb8c999be5b094619) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:42:42.876947 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b35f04d-ba91-4cd1-81f9-9ef4d0966ff0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:42:42.878383 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b35f04d-ba91-4cd1-81f9-9ef4d0966ff0 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:42:42.889723 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 603/1219] 10.222.0.27 () {56 vars in 1374 bytes} [Sun Sep 11 00:42:42 2022] GET /identity/v3/role_assignments?user.id=cdbfc2ccd73e4a5eb8c999be5b094619&scope.project.id=b504a5058bc6474c8cc077dfeec16799&role.id=0390f926e7ef4715ae18697033bde90e => generated 259 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:42:45.131818 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f2894f-934f-4997-84f8-4de80b2c1bb6 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:45.132361 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f2894f-934f-4997-84f8-4de80b2c1bb6 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:45.132466 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f2894f-934f-4997-84f8-4de80b2c1bb6 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:45.133746 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 617/1220] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:42:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:42:45.140414 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f91283-9819-47ab-badc-bc78859a0648 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:45.140890 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f91283-9819-47ab-badc-bc78859a0648 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:45.141104 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f91283-9819-47ab-badc-bc78859a0648 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:45.162737 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-56f91283-9819-47ab-badc-bc78859a0648 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:42:45.163952 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-56f91283-9819-47ab-badc-bc78859a0648 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:42:45.238009 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 604/1221] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:42:45 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:42:45.520541 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f2894f-934f-4997-84f8-4de80b2c1bb6 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:42:45.635097 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f2894f-934f-4997-84f8-4de80b2c1bb6 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:42:45.673626 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7910802c-b788-4a57-9c53-46c305bb2334 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:42:45.674470 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7910802c-b788-4a57-9c53-46c305bb2334 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:45.674849 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7910802c-b788-4a57-9c53-46c305bb2334 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:45.675158 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7910802c-b788-4a57-9c53-46c305bb2334 None admin] PATH_INFO: `/v3/roles/service` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:45.688863 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7910802c-b788-4a57-9c53-46c305bb2334 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:42:45.693889 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-7910802c-b788-4a57-9c53-46c305bb2334 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 11 00:42:45.693889 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:42:45.693889 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:42:45.693889 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:42:45.693889 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:42:45.693889 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:42:45.693889 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:42:45.693889 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:42:45.693889 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:42:45.693889 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:42:45.693889 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:42:45.693889 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:42:45.693889 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:42:45.693889 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:42:45.693889 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:42:45.693889 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:42:45.695176 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:42:45.695176 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:42:45.695176 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:42:45.695176 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:42:45.695176 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:42:45.695176 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:42:45.695176 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:42:45.695176 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:42:45.695176 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:42:45.695176 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:42:45.695176 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:42:45.695176 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:42:45.695176 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:42:45.695176 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:42:45.695176 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:42:45.695176 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:42:45.696351 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:42:45.696351 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:42:45.696351 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:42:45.696351 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:42:45.696351 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:42:45.696351 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:42:45.696351 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:42:45.696351 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:42:45.696351 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:42:45.696351 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:42:45.696351 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 11 00:42:45.696351 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:42:45.696351 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 618/1222] 10.222.0.27 () {56 vars in 1102 bytes} [Sun Sep 11 00:42:45 2022] GET /identity/v3/roles/service => generated 85 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:42:45.701198 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f91283-9819-47ab-badc-bc78859a0648 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:42:45.811590 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f91283-9819-47ab-badc-bc78859a0648 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:42:45.837843 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:42:45.849770 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122d12f8-2d94-4375-9a9d-3f47efcda11e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:42:45.850436 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122d12f8-2d94-4375-9a9d-3f47efcda11e None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:45.850712 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122d12f8-2d94-4375-9a9d-3f47efcda11e None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:45.851071 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122d12f8-2d94-4375-9a9d-3f47efcda11e None admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:45.851669 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-122d12f8-2d94-4375-9a9d-3f47efcda11e None admin] RBAC: Adding query filter params (name=service) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:42:45.851972 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-122d12f8-2d94-4375-9a9d-3f47efcda11e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:42:45.853325 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-122d12f8-2d94-4375-9a9d-3f47efcda11e None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:42:45.861700 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 605/1223] 10.222.0.27 () {56 vars in 1103 bytes} [Sun Sep 11 00:42:45 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:42:45.867738 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7910802c-b788-4a57-9c53-46c305bb2334 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:42:45.916000 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:42:45.982584 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7910802c-b788-4a57-9c53-46c305bb2334 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:42:46.022715 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e80563-0257-4eda-af07-3429b84fb463 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:42:46.023449 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e80563-0257-4eda-af07-3429b84fb463 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:46.023838 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e80563-0257-4eda-af07-3429b84fb463 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:46.024153 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e80563-0257-4eda-af07-3429b84fb463 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:46.037651 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86e80563-0257-4eda-af07-3429b84fb463 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:42:46.039471 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86e80563-0257-4eda-af07-3429b84fb463 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:42:46.056726 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-86e80563-0257-4eda-af07-3429b84fb463 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:42:46.056726 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:42:46.056726 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:42:46.056726 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:42:46.056726 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:42:46.056726 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:42:46.056726 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:42:46.056726 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:42:46.056726 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:42:46.056726 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:42:46.056726 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:42:46.056726 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:42:46.056726 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:42:46.056726 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:42:46.056726 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:42:46.056726 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:42:46.057934 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:42:46.057934 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:42:46.057934 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:42:46.057934 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:42:46.057934 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:42:46.057934 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:42:46.057934 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:42:46.057934 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:42:46.057934 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:42:46.057934 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:42:46.057934 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:42:46.057934 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:42:46.057934 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:42:46.057934 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:42:46.057934 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:42:46.057934 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:42:46.057934 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:42:46.058999 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:42:46.058999 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:42:46.058999 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:42:46.058999 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 619/1224] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:42:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:42:46.064250 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122d12f8-2d94-4375-9a9d-3f47efcda11e None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:42:46.183396 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122d12f8-2d94-4375-9a9d-3f47efcda11e None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:42:46.222741 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-348d0914-8f7e-4a12-a8ae-2e7368a969ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:42:46.223365 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348d0914-8f7e-4a12-a8ae-2e7368a969ab None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:46.223640 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348d0914-8f7e-4a12-a8ae-2e7368a969ab None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:46.223937 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348d0914-8f7e-4a12-a8ae-2e7368a969ab None admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:46.224538 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-348d0914-8f7e-4a12-a8ae-2e7368a969ab None admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:42:46.224862 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-348d0914-8f7e-4a12-a8ae-2e7368a969ab None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:42:46.226201 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-348d0914-8f7e-4a12-a8ae-2e7368a969ab None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:42:46.238716 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 606/1225] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:42:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:42:46.244652 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e80563-0257-4eda-af07-3429b84fb463 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:42:46.344331 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:42:46.352224 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e80563-0257-4eda-af07-3429b84fb463 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:42:46.388290 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a65bdf16-d270-432a-b86a-57b210a63ede None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:42:46.389139 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65bdf16-d270-432a-b86a-57b210a63ede None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:46.389549 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65bdf16-d270-432a-b86a-57b210a63ede None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:46.389995 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65bdf16-d270-432a-b86a-57b210a63ede None admin] PATH_INFO: `/v3/users/placement` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:46.397684 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a65bdf16-d270-432a-b86a-57b210a63ede None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:42:46.399163 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a65bdf16-d270-432a-b86a-57b210a63ede None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:42:46.413290 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-a65bdf16-d270-432a-b86a-57b210a63ede None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 11 00:42:46.413290 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:42:46.413290 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:42:46.413290 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:42:46.413290 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:42:46.413290 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:42:46.413290 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:42:46.413290 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:42:46.413290 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:42:46.413290 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:42:46.413290 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:42:46.413290 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:42:46.413290 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 11 00:42:46.413290 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 11 00:42:46.413290 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 11 00:42:46.413290 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 11 00:42:46.414426 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:42:46.414426 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:42:46.414426 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 11 00:42:46.414426 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:42:46.414426 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 11 00:42:46.414426 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:42:46.414426 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:42:46.414426 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:42:46.414426 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:42:46.414426 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:42:46.414426 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:42:46.414426 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:42:46.414426 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:42:46.414426 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:42:46.414426 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:42:46.414426 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:42:46.414426 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:42:46.414426 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:42:46.415544 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:42:46.415544 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:42:46.415544 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 11 00:42:46.415544 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 11 00:42:46.415544 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 11 00:42:46.415544 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 11 00:42:46.415544 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 11 00:42:46.415544 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 11 00:42:46.415544 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 11 00:42:46.415544 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:42:46.415544 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 620/1226] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:42:46 2022] GET /identity/v3/users/placement => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:42:46.420963 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-348d0914-8f7e-4a12-a8ae-2e7368a969ab None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:42:46.471081 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:42:46.535987 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-348d0914-8f7e-4a12-a8ae-2e7368a969ab None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:42:46.575759 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f94600c-b745-4714-af72-a85035b644f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:42:46.576971 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f94600c-b745-4714-af72-a85035b644f0 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:46.577285 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f94600c-b745-4714-af72-a85035b644f0 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:46.577593 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f94600c-b745-4714-af72-a85035b644f0 None admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:46.578497 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f94600c-b745-4714-af72-a85035b644f0 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:42:46.578999 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f94600c-b745-4714-af72-a85035b644f0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:42:46.580695 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f94600c-b745-4714-af72-a85035b644f0 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:42:46.601327 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 607/1227] 10.222.0.27 () {56 vars in 1143 bytes} [Sun Sep 11 00:42:46 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:42:46.607178 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a65bdf16-d270-432a-b86a-57b210a63ede None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:42:46.725454 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a65bdf16-d270-432a-b86a-57b210a63ede None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:42:46.762028 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-367a0a49-9782-4f35-87ea-9812f71ced9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:42:46.762671 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367a0a49-9782-4f35-87ea-9812f71ced9a None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:46.762968 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367a0a49-9782-4f35-87ea-9812f71ced9a None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:46.763242 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367a0a49-9782-4f35-87ea-9812f71ced9a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:46.775986 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-367a0a49-9782-4f35-87ea-9812f71ced9a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:42:46.777478 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-367a0a49-9782-4f35-87ea-9812f71ced9a None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:42:46.796695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-367a0a49-9782-4f35-87ea-9812f71ced9a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:42:46.796695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:42:46.796695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:42:46.796695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:42:46.796695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:42:46.796695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:42:46.796695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:42:46.796695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:42:46.796695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:42:46.796695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:42:46.796695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:42:46.796695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:42:46.796695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:42:46.796695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:42:46.796695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:42:46.796695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:42:46.797828 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:42:46.797828 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:42:46.797828 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:42:46.797828 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:42:46.797828 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:42:46.797828 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:42:46.797828 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:42:46.797828 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:42:46.797828 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:42:46.797828 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:42:46.797828 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:42:46.797828 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:42:46.797828 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:42:46.797828 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:42:46.797828 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:42:46.797828 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:42:46.797828 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:42:46.798819 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:42:46.798819 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:42:46.798819 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:42:46.798819 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 621/1228] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:42:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:42:46.803623 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f94600c-b745-4714-af72-a85035b644f0 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:42:46.911919 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f94600c-b745-4714-af72-a85035b644f0 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:42:46.914127 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:42:46.947743 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f93cae-613d-4d7b-b5a4-8c9a9f28c798 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:42:46.948339 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f93cae-613d-4d7b-b5a4-8c9a9f28c798 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:46.948611 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f93cae-613d-4d7b-b5a4-8c9a9f28c798 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:46.948883 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f93cae-613d-4d7b-b5a4-8c9a9f28c798 None admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:46.949514 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f93cae-613d-4d7b-b5a4-8c9a9f28c798 None admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:42:46.949810 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f93cae-613d-4d7b-b5a4-8c9a9f28c798 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:42:46.951209 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f93cae-613d-4d7b-b5a4-8c9a9f28c798 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:42:46.969225 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 608/1229] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:42:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:42:46.970440 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-367a0a49-9782-4f35-87ea-9812f71ced9a None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:42:46.980519 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:42:47.081559 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-367a0a49-9782-4f35-87ea-9812f71ced9a None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:42:47.120191 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f2e9e7-6f9f-4644-b227-822c067cc819 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:42:47.120900 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f2e9e7-6f9f-4644-b227-822c067cc819 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:47.121241 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f2e9e7-6f9f-4644-b227-822c067cc819 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:47.121536 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f2e9e7-6f9f-4644-b227-822c067cc819 None admin] PATH_INFO: `/v3/projects/service` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:47.128673 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0f2e9e7-6f9f-4644-b227-822c067cc819 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:42:47.130116 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0f2e9e7-6f9f-4644-b227-822c067cc819 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:42:47.141572 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-a0f2e9e7-6f9f-4644-b227-822c067cc819 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:42:47.141572 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:42:47.141572 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:42:47.141572 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:42:47.141572 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:42:47.141572 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:42:47.141572 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:42:47.141572 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:42:47.141572 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:42:47.141572 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:42:47.141572 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:42:47.141572 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:42:47.141572 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 11 00:42:47.141572 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 11 00:42:47.141572 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 11 00:42:47.142675 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 11 00:42:47.142675 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:42:47.142675 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:42:47.142675 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:42:47.142675 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:42:47.142675 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:42:47.142675 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:42:47.142675 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:42:47.142675 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:42:47.142675 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:42:47.142675 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:42:47.142675 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:42:47.142675 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:42:47.142675 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:42:47.142675 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:42:47.142675 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:42:47.142675 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:42:47.143618 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 11 00:42:47.143618 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 11 00:42:47.143618 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 11 00:42:47.143618 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 11 00:42:47.143618 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 11 00:42:47.143618 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 11 00:42:47.143618 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:42:47.143618 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:42:47.143618 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 622/1230] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:42:46 2022] GET /identity/v3/projects/service => generated 88 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:42:47.148715 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f93cae-613d-4d7b-b5a4-8c9a9f28c798 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:42:47.264828 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f93cae-613d-4d7b-b5a4-8c9a9f28c798 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:42:47.303145 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0915a532-f61c-43bd-9b61-c2e90d7f4f41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:42:47.303752 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0915a532-f61c-43bd-9b61-c2e90d7f4f41 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:47.304049 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0915a532-f61c-43bd-9b61-c2e90d7f4f41 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:47.304397 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0915a532-f61c-43bd-9b61-c2e90d7f4f41 None admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:47.305468 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0915a532-f61c-43bd-9b61-c2e90d7f4f41 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:42:47.305777 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0915a532-f61c-43bd-9b61-c2e90d7f4f41 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:42:47.307306 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0915a532-f61c-43bd-9b61-c2e90d7f4f41 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:42:47.319415 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 609/1231] 10.222.0.27 () {56 vars in 1148 bytes} [Sun Sep 11 00:42:47 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:42:47.325616 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f2e9e7-6f9f-4644-b227-822c067cc819 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:42:47.397070 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:42:47.433797 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f2e9e7-6f9f-4644-b227-822c067cc819 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:42:47.469471 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0710ac44-f2b0-442d-b2da-9df279c5cd96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:42:47.470154 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0710ac44-f2b0-442d-b2da-9df279c5cd96 None admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:47.470433 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0710ac44-f2b0-442d-b2da-9df279c5cd96 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:47.470706 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0710ac44-f2b0-442d-b2da-9df279c5cd96 None admin] PATH_INFO: `/v3/projects/b504a5058bc6474c8cc077dfeec16799/users/cdbfc2ccd73e4a5eb8c999be5b094619/roles/0390f926e7ef4715ae18697033bde90e` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:47.487302 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0710ac44-f2b0-442d-b2da-9df279c5cd96 None admin] RBAC: Authorizing `identity:create_grant(project_id=b504a5058bc6474c8cc077dfeec16799, user_id=cdbfc2ccd73e4a5eb8c999be5b094619, role_id=0390f926e7ef4715ae18697033bde90e)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:42:47.488852 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0710ac44-f2b0-442d-b2da-9df279c5cd96 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:42:47.504307 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 623/1232] 10.222.0.27 () {58 vars in 1439 bytes} [Sun Sep 11 00:42:47 2022] PUT /identity/v3/projects/b504a5058bc6474c8cc077dfeec16799/users/cdbfc2ccd73e4a5eb8c999be5b094619/roles/0390f926e7ef4715ae18697033bde90e => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:42:49.736053 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e94b678-5cbb-4a24-b2ec-3d402245a021 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:49.736657 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e94b678-5cbb-4a24-b2ec-3d402245a021 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:49.736657 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e94b678-5cbb-4a24-b2ec-3d402245a021 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:49.737998 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 610/1233] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:42:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:42:49.743953 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be62bf05-f850-45f5-8ead-6dda8b1cae68 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:49.744444 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be62bf05-f850-45f5-8ead-6dda8b1cae68 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:49.744587 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be62bf05-f850-45f5-8ead-6dda8b1cae68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:49.764299 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-be62bf05-f850-45f5-8ead-6dda8b1cae68 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:42:49.765620 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-be62bf05-f850-45f5-8ead-6dda8b1cae68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:42:49.837316 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 624/1234] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:42:49 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:42:50.133401 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e94b678-5cbb-4a24-b2ec-3d402245a021 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:42:50.167303 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:42:50.258243 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e94b678-5cbb-4a24-b2ec-3d402245a021 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:42:50.301731 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c037c4-63c1-432f-9193-aa191a9275c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:42:50.302354 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c037c4-63c1-432f-9193-aa191a9275c8 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:50.302781 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c037c4-63c1-432f-9193-aa191a9275c8 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:50.303106 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c037c4-63c1-432f-9193-aa191a9275c8 None admin] PATH_INFO: `/v3/roles/service` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:50.314481 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3c037c4-63c1-432f-9193-aa191a9275c8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:42:50.319279 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-f3c037c4-63c1-432f-9193-aa191a9275c8 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 11 00:42:50.319279 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:42:50.319279 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:42:50.319279 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:42:50.319279 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:42:50.319279 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:42:50.319279 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:42:50.319279 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:42:50.319279 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:42:50.319279 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:42:50.319279 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:42:50.319279 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:42:50.319279 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:42:50.319279 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:42:50.319279 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:42:50.319279 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:42:50.320471 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:42:50.320471 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:42:50.320471 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:42:50.320471 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:42:50.320471 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:42:50.320471 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:42:50.320471 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:42:50.320471 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:42:50.320471 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:42:50.320471 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:42:50.320471 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:42:50.320471 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:42:50.320471 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:42:50.320471 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:42:50.320471 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:42:50.320471 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:42:50.321743 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:42:50.321743 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:42:50.321743 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:42:50.321743 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:42:50.321743 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:42:50.321743 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:42:50.321743 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:42:50.321743 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:42:50.321743 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:42:50.321743 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:42:50.321743 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 11 00:42:50.321743 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:42:50.321743 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 611/1235] 10.222.0.27 () {56 vars in 1102 bytes} [Sun Sep 11 00:42:50 2022] GET /identity/v3/roles/service => generated 85 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:42:50.326961 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be62bf05-f850-45f5-8ead-6dda8b1cae68 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:42:50.382201 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:42:50.445381 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be62bf05-f850-45f5-8ead-6dda8b1cae68 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:42:50.481873 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc182737-11e6-4956-a760-efc1faa4981f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:42:50.482475 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc182737-11e6-4956-a760-efc1faa4981f None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:50.482871 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc182737-11e6-4956-a760-efc1faa4981f None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:50.483189 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc182737-11e6-4956-a760-efc1faa4981f None admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:50.483919 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc182737-11e6-4956-a760-efc1faa4981f None admin] RBAC: Adding query filter params (name=service) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:42:50.484202 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc182737-11e6-4956-a760-efc1faa4981f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:42:50.485731 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc182737-11e6-4956-a760-efc1faa4981f None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:42:50.494501 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 625/1236] 10.222.0.27 () {56 vars in 1103 bytes} [Sun Sep 11 00:42:50 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:42:50.501102 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c037c4-63c1-432f-9193-aa191a9275c8 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:42:50.602149 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:42:50.620236 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c037c4-63c1-432f-9193-aa191a9275c8 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:42:50.658743 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b24993-549b-4f9f-8714-ef916090cd68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:42:50.659306 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b24993-549b-4f9f-8714-ef916090cd68 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:50.659582 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b24993-549b-4f9f-8714-ef916090cd68 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:50.659850 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b24993-549b-4f9f-8714-ef916090cd68 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:50.670312 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3b24993-549b-4f9f-8714-ef916090cd68 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:42:50.672069 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3b24993-549b-4f9f-8714-ef916090cd68 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:42:50.689281 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-b3b24993-549b-4f9f-8714-ef916090cd68 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:42:50.689281 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:42:50.689281 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:42:50.689281 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:42:50.689281 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:42:50.689281 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:42:50.689281 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:42:50.689281 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:42:50.689281 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:42:50.689281 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:42:50.689281 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:42:50.689281 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:42:50.689281 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:42:50.689281 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:42:50.689281 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:42:50.689281 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:42:50.690507 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:42:50.690507 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:42:50.690507 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:42:50.690507 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:42:50.690507 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:42:50.690507 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:42:50.690507 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:42:50.690507 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:42:50.690507 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:42:50.690507 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:42:50.690507 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:42:50.690507 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:42:50.690507 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:42:50.690507 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:42:50.690507 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:42:50.690507 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:42:50.690507 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:42:50.691678 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:42:50.691678 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:42:50.691678 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:42:50.691678 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 612/1237] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:42:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:42:50.696797 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc182737-11e6-4956-a760-efc1faa4981f None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:42:50.812434 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc182737-11e6-4956-a760-efc1faa4981f None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:42:50.822521 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:42:50.852500 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ecbb8cd-ee45-4889-ac99-8411c261a936 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:42:50.853312 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ecbb8cd-ee45-4889-ac99-8411c261a936 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:50.853715 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ecbb8cd-ee45-4889-ac99-8411c261a936 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:50.854108 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ecbb8cd-ee45-4889-ac99-8411c261a936 None admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:50.854832 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ecbb8cd-ee45-4889-ac99-8411c261a936 None admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:42:50.855208 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ecbb8cd-ee45-4889-ac99-8411c261a936 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:42:50.856864 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ecbb8cd-ee45-4889-ac99-8411c261a936 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:42:50.872399 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 626/1238] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:42:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:42:50.879182 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b24993-549b-4f9f-8714-ef916090cd68 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:42:51.006223 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b24993-549b-4f9f-8714-ef916090cd68 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:42:51.044028 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d132f6-8703-4048-9f63-da6e287c11f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:42:51.044660 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d132f6-8703-4048-9f63-da6e287c11f4 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:51.044937 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d132f6-8703-4048-9f63-da6e287c11f4 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:51.045773 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d132f6-8703-4048-9f63-da6e287c11f4 None admin] PATH_INFO: `/v3/users/placement` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:51.049104 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:42:51.056116 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d132f6-8703-4048-9f63-da6e287c11f4 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:42:51.057612 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d132f6-8703-4048-9f63-da6e287c11f4 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:42:51.070002 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-66d132f6-8703-4048-9f63-da6e287c11f4 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 11 00:42:51.070002 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:42:51.070002 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:42:51.070002 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:42:51.070002 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:42:51.070002 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:42:51.070002 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:42:51.070002 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:42:51.070002 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:42:51.070002 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:42:51.070002 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:42:51.070002 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:42:51.070002 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 11 00:42:51.070002 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 11 00:42:51.070002 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 11 00:42:51.070002 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 11 00:42:51.071250 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:42:51.071250 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:42:51.071250 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 11 00:42:51.071250 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:42:51.071250 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 11 00:42:51.071250 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:42:51.071250 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:42:51.071250 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:42:51.071250 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:42:51.071250 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:42:51.071250 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:42:51.071250 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:42:51.071250 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:42:51.071250 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:42:51.071250 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:42:51.071250 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:42:51.071250 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:42:51.071250 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:42:51.072444 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:42:51.072444 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:42:51.072444 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 11 00:42:51.072444 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 11 00:42:51.072444 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 11 00:42:51.072444 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 11 00:42:51.072444 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 11 00:42:51.072444 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 11 00:42:51.072444 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 11 00:42:51.072444 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:42:51.072444 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 613/1239] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:42:50 2022] GET /identity/v3/users/placement => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:42:51.078352 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ecbb8cd-ee45-4889-ac99-8411c261a936 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:42:51.189446 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ecbb8cd-ee45-4889-ac99-8411c261a936 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:42:51.225821 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f3376d7-5068-46ce-b949-927730b2d08d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:42:51.226494 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f3376d7-5068-46ce-b949-927730b2d08d None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:51.226757 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f3376d7-5068-46ce-b949-927730b2d08d None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:51.227230 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f3376d7-5068-46ce-b949-927730b2d08d None admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:51.228216 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f3376d7-5068-46ce-b949-927730b2d08d None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:42:51.228653 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f3376d7-5068-46ce-b949-927730b2d08d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:42:51.230355 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f3376d7-5068-46ce-b949-927730b2d08d None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:42:51.252315 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 627/1240] 10.222.0.27 () {56 vars in 1143 bytes} [Sun Sep 11 00:42:51 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:42:51.258384 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d132f6-8703-4048-9f63-da6e287c11f4 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:42:51.370890 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d132f6-8703-4048-9f63-da6e287c11f4 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:42:51.410986 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf7cabc-2271-4b04-a37a-27858b7d5d31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:42:51.411675 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf7cabc-2271-4b04-a37a-27858b7d5d31 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:51.412022 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf7cabc-2271-4b04-a37a-27858b7d5d31 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:51.412271 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf7cabc-2271-4b04-a37a-27858b7d5d31 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:51.491807 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:42:51.500920 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cf7cabc-2271-4b04-a37a-27858b7d5d31 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:42:51.502481 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cf7cabc-2271-4b04-a37a-27858b7d5d31 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:42:51.630430 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 614/1241] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:42:51 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 11 00:42:51.636395 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f3376d7-5068-46ce-b949-927730b2d08d None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:42:51.671104 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:42:51.763556 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f3376d7-5068-46ce-b949-927730b2d08d None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:42:51.807203 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-784cb452-5fe0-4071-a77f-b39aaf6e01a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:42:51.807908 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784cb452-5fe0-4071-a77f-b39aaf6e01a7 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:51.808285 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784cb452-5fe0-4071-a77f-b39aaf6e01a7 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:51.808563 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784cb452-5fe0-4071-a77f-b39aaf6e01a7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:51.819739 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-784cb452-5fe0-4071-a77f-b39aaf6e01a7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:42:51.821215 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-784cb452-5fe0-4071-a77f-b39aaf6e01a7 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:42:51.837744 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-784cb452-5fe0-4071-a77f-b39aaf6e01a7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:42:51.837744 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:42:51.837744 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:42:51.837744 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:42:51.837744 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:42:51.837744 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:42:51.837744 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:42:51.837744 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:42:51.837744 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:42:51.837744 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:42:51.837744 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:42:51.837744 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:42:51.837744 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:42:51.837744 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:42:51.837744 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:42:51.837744 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:42:51.838750 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:42:51.838750 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:42:51.838750 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:42:51.838750 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:42:51.838750 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:42:51.838750 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:42:51.838750 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:42:51.838750 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:42:51.838750 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:42:51.838750 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:42:51.838750 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:42:51.838750 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:42:51.838750 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:42:51.838750 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:42:51.838750 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:42:51.838750 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:42:51.838750 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:42:51.848270 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:42:51.848270 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:42:51.848270 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:42:51.848270 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 628/1242] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:42:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:42:51.848722 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf7cabc-2271-4b04-a37a-27858b7d5d31 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:42:51.954262 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:42:51.961651 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf7cabc-2271-4b04-a37a-27858b7d5d31 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:42:52.000530 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3614c0d-7d57-4cf1-a658-ee7b1f1592c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:42:52.001289 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3614c0d-7d57-4cf1-a658-ee7b1f1592c9 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:52.001687 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3614c0d-7d57-4cf1-a658-ee7b1f1592c9 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:52.001960 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3614c0d-7d57-4cf1-a658-ee7b1f1592c9 None admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:52.002630 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3614c0d-7d57-4cf1-a658-ee7b1f1592c9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:42:52.002966 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3614c0d-7d57-4cf1-a658-ee7b1f1592c9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:42:52.004355 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3614c0d-7d57-4cf1-a658-ee7b1f1592c9 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:42:52.017919 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 615/1243] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:42:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:42:52.023742 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-784cb452-5fe0-4071-a77f-b39aaf6e01a7 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:42:52.113616 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:42:52.141613 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-784cb452-5fe0-4071-a77f-b39aaf6e01a7 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:42:52.180388 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-812111e9-31e7-4d29-aded-60ffd2ccf6ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:42:52.181139 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812111e9-31e7-4d29-aded-60ffd2ccf6ba None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:52.181434 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812111e9-31e7-4d29-aded-60ffd2ccf6ba None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:52.181733 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812111e9-31e7-4d29-aded-60ffd2ccf6ba None admin] PATH_INFO: `/v3/projects/service` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:52.191475 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-812111e9-31e7-4d29-aded-60ffd2ccf6ba None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:42:52.192137 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-812111e9-31e7-4d29-aded-60ffd2ccf6ba None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:42:52.204726 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-812111e9-31e7-4d29-aded-60ffd2ccf6ba None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:42:52.204726 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:42:52.204726 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:42:52.204726 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:42:52.204726 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:42:52.204726 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:42:52.204726 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:42:52.204726 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:42:52.204726 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:42:52.204726 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:42:52.204726 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:42:52.204726 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:42:52.204726 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 11 00:42:52.204726 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 11 00:42:52.204726 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 11 00:42:52.205681 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 11 00:42:52.205681 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:42:52.205681 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:42:52.205681 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:42:52.205681 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:42:52.205681 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:42:52.205681 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:42:52.205681 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:42:52.205681 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:42:52.205681 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:42:52.205681 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:42:52.205681 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:42:52.205681 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:42:52.205681 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:42:52.205681 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:42:52.205681 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:42:52.205681 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:42:52.206682 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 11 00:42:52.206682 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 11 00:42:52.206682 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 11 00:42:52.206682 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 11 00:42:52.206682 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 11 00:42:52.206682 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 11 00:42:52.206682 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:42:52.206682 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:42:52.206682 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 629/1244] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:42:52 2022] GET /identity/v3/projects/service => generated 88 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:42:52.212338 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3614c0d-7d57-4cf1-a658-ee7b1f1592c9 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:42:52.332771 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3614c0d-7d57-4cf1-a658-ee7b1f1592c9 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:42:52.368976 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f71eae12-94c4-4fd9-a6a2-fb2892d09429 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:42:52.369641 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f71eae12-94c4-4fd9-a6a2-fb2892d09429 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:52.369990 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f71eae12-94c4-4fd9-a6a2-fb2892d09429 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:52.370270 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f71eae12-94c4-4fd9-a6a2-fb2892d09429 None admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:52.371023 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f71eae12-94c4-4fd9-a6a2-fb2892d09429 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:42:52.371365 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f71eae12-94c4-4fd9-a6a2-fb2892d09429 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:42:52.372914 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f71eae12-94c4-4fd9-a6a2-fb2892d09429 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:42:52.386730 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 616/1245] 10.222.0.27 () {56 vars in 1148 bytes} [Sun Sep 11 00:42:52 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:42:52.394135 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-812111e9-31e7-4d29-aded-60ffd2ccf6ba None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:42:52.528231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-812111e9-31e7-4d29-aded-60ffd2ccf6ba None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:42:52.570322 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f22ecba-27ba-4c8e-8974-72b41701aedf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:42:52.571028 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f22ecba-27ba-4c8e-8974-72b41701aedf None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:52.571295 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f22ecba-27ba-4c8e-8974-72b41701aedf None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:52.571561 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f22ecba-27ba-4c8e-8974-72b41701aedf None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:52.572284 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f22ecba-27ba-4c8e-8974-72b41701aedf None admin] RBAC: Adding query filter params (role.id=0390f926e7ef4715ae18697033bde90e, scope.project.id=b504a5058bc6474c8cc077dfeec16799, user.id=cdbfc2ccd73e4a5eb8c999be5b094619) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:42:52.572587 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f22ecba-27ba-4c8e-8974-72b41701aedf None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:42:52.574237 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f22ecba-27ba-4c8e-8974-72b41701aedf None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:42:52.581924 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 630/1246] 10.222.0.27 () {56 vars in 1374 bytes} [Sun Sep 11 00:42:52 2022] GET /identity/v3/role_assignments?user.id=cdbfc2ccd73e4a5eb8c999be5b094619&scope.project.id=b504a5058bc6474c8cc077dfeec16799&role.id=0390f926e7ef4715ae18697033bde90e => generated 608 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:42:54.921443 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d373771b-6ecc-4430-8327-de40c8747f7a None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:54.922012 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d373771b-6ecc-4430-8327-de40c8747f7a None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:54.922012 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d373771b-6ecc-4430-8327-de40c8747f7a None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:54.923340 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 617/1247] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:42:54 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:42:54.928435 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213da3f1-ddbb-4420-a855-ee5f5def9ff1 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:54.928884 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213da3f1-ddbb-4420-a855-ee5f5def9ff1 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:54.928989 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213da3f1-ddbb-4420-a855-ee5f5def9ff1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:54.935192 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:42:54.948816 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-213da3f1-ddbb-4420-a855-ee5f5def9ff1 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:42:54.949993 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-213da3f1-ddbb-4420-a855-ee5f5def9ff1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:42:55.032343 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 631/1248] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:42:54 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 11 00:42:55.321735 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d373771b-6ecc-4430-8327-de40c8747f7a None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:42:55.341504 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:42:55.437729 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d373771b-6ecc-4430-8327-de40c8747f7a None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:42:55.479293 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c91d916-c7f8-4484-b76a-2783b4d9a2fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:42:55.480037 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c91d916-c7f8-4484-b76a-2783b4d9a2fa None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:55.480379 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c91d916-c7f8-4484-b76a-2783b4d9a2fa None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:55.480697 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c91d916-c7f8-4484-b76a-2783b4d9a2fa None admin] PATH_INFO: `/v3/roles/admin` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:55.494449 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c91d916-c7f8-4484-b76a-2783b4d9a2fa None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:42:55.501300 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-8c91d916-c7f8-4484-b76a-2783b4d9a2fa None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:42:55.501300 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:42:55.501300 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:42:55.501300 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:42:55.501300 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:42:55.501300 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:42:55.501300 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:42:55.501300 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:42:55.501300 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:42:55.501300 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:42:55.501300 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:42:55.501300 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:42:55.501300 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:42:55.501300 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:42:55.501300 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:42:55.501300 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:42:55.503310 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:42:55.503310 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:42:55.503310 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:42:55.503310 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:42:55.503310 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:42:55.503310 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:42:55.503310 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:42:55.503310 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:42:55.503310 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:42:55.503310 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:42:55.503310 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:42:55.503310 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:42:55.503310 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:42:55.503310 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:42:55.503310 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:42:55.503310 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:42:55.504288 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:42:55.504288 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:42:55.504288 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:42:55.504288 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:42:55.504288 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:42:55.504288 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:42:55.504288 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:42:55.504288 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:42:55.504288 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:42:55.504288 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:42:55.504288 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:42:55.504288 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:42:55.504288 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 618/1249] 10.222.0.27 () {56 vars in 1096 bytes} [Sun Sep 11 00:42:55 2022] GET /identity/v3/roles/admin => generated 83 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:42:55.510273 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-213da3f1-ddbb-4420-a855-ee5f5def9ff1 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:42:55.643800 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-213da3f1-ddbb-4420-a855-ee5f5def9ff1 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:42:55.680844 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:42:55.688345 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21800132-a536-4a1a-bebd-0b00ca74bc5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:42:55.688977 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21800132-a536-4a1a-bebd-0b00ca74bc5a None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:55.689269 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21800132-a536-4a1a-bebd-0b00ca74bc5a None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:55.689527 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21800132-a536-4a1a-bebd-0b00ca74bc5a None admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:55.690197 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21800132-a536-4a1a-bebd-0b00ca74bc5a None admin] RBAC: Adding query filter params (name=admin) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:42:55.690502 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21800132-a536-4a1a-bebd-0b00ca74bc5a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:42:55.691909 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21800132-a536-4a1a-bebd-0b00ca74bc5a None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:42:55.700460 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 632/1250] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:42:55 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:42:55.706839 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c91d916-c7f8-4484-b76a-2783b4d9a2fa None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:42:55.792167 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:42:55.816378 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c91d916-c7f8-4484-b76a-2783b4d9a2fa None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:42:55.853518 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d085e12-1a9f-47bc-8a37-b9881a45b3e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:42:55.854157 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d085e12-1a9f-47bc-8a37-b9881a45b3e2 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:55.854438 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d085e12-1a9f-47bc-8a37-b9881a45b3e2 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:55.854708 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d085e12-1a9f-47bc-8a37-b9881a45b3e2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:55.866453 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d085e12-1a9f-47bc-8a37-b9881a45b3e2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:42:55.867965 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d085e12-1a9f-47bc-8a37-b9881a45b3e2 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:42:55.883644 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-9d085e12-1a9f-47bc-8a37-b9881a45b3e2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:42:55.883644 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:42:55.883644 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:42:55.883644 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:42:55.883644 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:42:55.883644 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:42:55.883644 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:42:55.883644 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:42:55.883644 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:42:55.883644 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:42:55.883644 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:42:55.883644 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:42:55.883644 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:42:55.883644 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:42:55.883644 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:42:55.883644 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:42:55.884744 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:42:55.884744 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:42:55.884744 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:42:55.884744 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:42:55.884744 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:42:55.884744 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:42:55.884744 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:42:55.884744 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:42:55.884744 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:42:55.884744 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:42:55.884744 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:42:55.884744 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:42:55.884744 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:42:55.884744 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:42:55.884744 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:42:55.884744 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:42:55.884744 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:42:55.885690 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:42:55.885690 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:42:55.885690 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:42:55.885690 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 619/1251] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:42:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:42:55.891413 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21800132-a536-4a1a-bebd-0b00ca74bc5a None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:42:55.996181 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21800132-a536-4a1a-bebd-0b00ca74bc5a None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:42:56.032886 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b71fab9-8e25-4c45-8dec-391d853e9e1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:42:56.033776 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b71fab9-8e25-4c45-8dec-391d853e9e1d None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:56.034051 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b71fab9-8e25-4c45-8dec-391d853e9e1d None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:56.034318 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b71fab9-8e25-4c45-8dec-391d853e9e1d None admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:56.034999 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b71fab9-8e25-4c45-8dec-391d853e9e1d None admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:42:56.035304 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b71fab9-8e25-4c45-8dec-391d853e9e1d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:42:56.036707 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b71fab9-8e25-4c45-8dec-391d853e9e1d None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:42:56.050110 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 633/1252] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:42:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:42:56.055991 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d085e12-1a9f-47bc-8a37-b9881a45b3e2 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:42:56.163960 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d085e12-1a9f-47bc-8a37-b9881a45b3e2 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:42:56.191083 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:42:56.199336 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd007ebc-7a17-48e2-8a34-9d5134fae8ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:42:56.199961 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd007ebc-7a17-48e2-8a34-9d5134fae8ae None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:56.200232 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd007ebc-7a17-48e2-8a34-9d5134fae8ae None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:56.200497 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd007ebc-7a17-48e2-8a34-9d5134fae8ae None admin] PATH_INFO: `/v3/users/placement` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:56.208666 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd007ebc-7a17-48e2-8a34-9d5134fae8ae None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:42:56.210107 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd007ebc-7a17-48e2-8a34-9d5134fae8ae None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:42:56.222246 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-bd007ebc-7a17-48e2-8a34-9d5134fae8ae None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 11 00:42:56.222246 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:42:56.222246 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:42:56.222246 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:42:56.222246 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:42:56.222246 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:42:56.222246 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:42:56.222246 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:42:56.222246 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:42:56.222246 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:42:56.222246 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:42:56.222246 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:42:56.222246 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 11 00:42:56.222246 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 11 00:42:56.222246 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 11 00:42:56.222246 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 11 00:42:56.223459 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:42:56.223459 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:42:56.223459 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 11 00:42:56.223459 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:42:56.223459 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 11 00:42:56.223459 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:42:56.223459 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:42:56.223459 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:42:56.223459 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:42:56.223459 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:42:56.223459 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:42:56.223459 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:42:56.223459 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:42:56.223459 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:42:56.223459 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:42:56.223459 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:42:56.223459 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:42:56.223459 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:42:56.224640 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:42:56.224640 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:42:56.224640 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 11 00:42:56.224640 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 11 00:42:56.224640 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 11 00:42:56.224640 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 11 00:42:56.224640 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 11 00:42:56.224640 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 11 00:42:56.224640 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 11 00:42:56.224640 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:42:56.224640 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 620/1253] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:42:56 2022] GET /identity/v3/users/placement => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 11 00:42:56.229196 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b71fab9-8e25-4c45-8dec-391d853e9e1d None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:42:56.286264 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:42:56.333435 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b71fab9-8e25-4c45-8dec-391d853e9e1d None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:42:56.368803 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a77f031c-241b-4128-99fd-3f144e5f62e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:42:56.369424 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a77f031c-241b-4128-99fd-3f144e5f62e5 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:56.369706 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a77f031c-241b-4128-99fd-3f144e5f62e5 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:56.369974 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a77f031c-241b-4128-99fd-3f144e5f62e5 None admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:56.370705 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a77f031c-241b-4128-99fd-3f144e5f62e5 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:42:56.371042 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a77f031c-241b-4128-99fd-3f144e5f62e5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:42:56.372471 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a77f031c-241b-4128-99fd-3f144e5f62e5 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:42:56.390209 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 634/1254] 10.222.0.27 () {56 vars in 1143 bytes} [Sun Sep 11 00:42:56 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:42:56.396051 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd007ebc-7a17-48e2-8a34-9d5134fae8ae None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:42:56.497966 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd007ebc-7a17-48e2-8a34-9d5134fae8ae None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:42:56.536396 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-234d77c6-4227-4e2f-a8a4-7904eb5c8b3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:42:56.536932 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234d77c6-4227-4e2f-a8a4-7904eb5c8b3b None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:56.537208 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234d77c6-4227-4e2f-a8a4-7904eb5c8b3b None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:56.537478 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234d77c6-4227-4e2f-a8a4-7904eb5c8b3b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:56.591120 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:42:56.604150 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-234d77c6-4227-4e2f-a8a4-7904eb5c8b3b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:42:56.605592 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-234d77c6-4227-4e2f-a8a4-7904eb5c8b3b None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:42:56.700724 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 621/1255] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:42:56 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 11 00:42:56.707308 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a77f031c-241b-4128-99fd-3f144e5f62e5 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:42:56.803560 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a77f031c-241b-4128-99fd-3f144e5f62e5 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:42:56.816121 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:42:56.837283 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b6df1ed-340c-46a8-aed1-98626109d810 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:42:56.837898 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6df1ed-340c-46a8-aed1-98626109d810 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:56.838174 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6df1ed-340c-46a8-aed1-98626109d810 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:56.838442 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6df1ed-340c-46a8-aed1-98626109d810 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:56.848682 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b6df1ed-340c-46a8-aed1-98626109d810 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:42:56.850096 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b6df1ed-340c-46a8-aed1-98626109d810 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:42:56.864875 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-2b6df1ed-340c-46a8-aed1-98626109d810 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:42:56.864875 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:42:56.864875 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:42:56.864875 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:42:56.864875 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:42:56.864875 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:42:56.864875 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:42:56.864875 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:42:56.864875 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:42:56.864875 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:42:56.864875 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:42:56.864875 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:42:56.864875 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:42:56.864875 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:42:56.864875 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:42:56.864875 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:42:56.870481 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:42:56.870481 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:42:56.870481 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:42:56.870481 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:42:56.870481 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:42:56.870481 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:42:56.870481 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:42:56.870481 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:42:56.870481 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:42:56.870481 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:42:56.870481 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:42:56.870481 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:42:56.870481 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:42:56.870481 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:42:56.870481 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:42:56.870481 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:42:56.870481 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:42:56.871655 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:42:56.871655 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:42:56.871655 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:42:56.871655 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 635/1256] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:42:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:42:56.871940 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-234d77c6-4227-4e2f-a8a4-7904eb5c8b3b None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:42:56.956640 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:42:56.969701 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-234d77c6-4227-4e2f-a8a4-7904eb5c8b3b None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:42:57.004131 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-202f1797-d99c-4305-8fb9-5470c8b109bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:42:57.004717 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202f1797-d99c-4305-8fb9-5470c8b109bc None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:57.004992 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202f1797-d99c-4305-8fb9-5470c8b109bc None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:57.005261 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202f1797-d99c-4305-8fb9-5470c8b109bc None admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:57.005892 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-202f1797-d99c-4305-8fb9-5470c8b109bc None admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:42:57.006189 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-202f1797-d99c-4305-8fb9-5470c8b109bc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:42:57.007603 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-202f1797-d99c-4305-8fb9-5470c8b109bc None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:42:57.019087 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 622/1257] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:42:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:42:57.024636 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b6df1ed-340c-46a8-aed1-98626109d810 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:42:57.121577 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b6df1ed-340c-46a8-aed1-98626109d810 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:42:57.154050 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b25bcc-dab1-4314-b384-3da2c325b89b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:42:57.154676 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b25bcc-dab1-4314-b384-3da2c325b89b None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:57.154982 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b25bcc-dab1-4314-b384-3da2c325b89b None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:57.155256 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b25bcc-dab1-4314-b384-3da2c325b89b None admin] PATH_INFO: `/v3/projects/service` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:57.161917 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b25bcc-dab1-4314-b384-3da2c325b89b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:42:57.163397 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b25bcc-dab1-4314-b384-3da2c325b89b None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:42:57.175877 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-e1b25bcc-dab1-4314-b384-3da2c325b89b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:42:57.175877 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:42:57.175877 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:42:57.175877 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:42:57.175877 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:42:57.175877 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:42:57.175877 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:42:57.175877 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:42:57.175877 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:42:57.175877 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:42:57.175877 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:42:57.175877 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:42:57.175877 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 11 00:42:57.175877 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 11 00:42:57.175877 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 11 00:42:57.176960 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 11 00:42:57.176960 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:42:57.176960 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:42:57.176960 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:42:57.176960 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:42:57.176960 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:42:57.176960 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:42:57.176960 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:42:57.176960 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:42:57.176960 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:42:57.176960 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:42:57.176960 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:42:57.176960 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:42:57.176960 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:42:57.176960 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:42:57.176960 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:42:57.176960 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:42:57.177900 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 11 00:42:57.177900 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 11 00:42:57.177900 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 11 00:42:57.177900 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 11 00:42:57.177900 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 11 00:42:57.177900 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 11 00:42:57.177900 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:42:57.177900 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:42:57.177900 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 636/1258] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:42:57 2022] GET /identity/v3/projects/service => generated 88 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:42:57.182980 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-202f1797-d99c-4305-8fb9-5470c8b109bc None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:42:57.279715 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-202f1797-d99c-4305-8fb9-5470c8b109bc None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:42:57.312659 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58caf976-4b5f-4288-a1cc-4e434bad3f0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:42:57.313238 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58caf976-4b5f-4288-a1cc-4e434bad3f0a None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:57.313510 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58caf976-4b5f-4288-a1cc-4e434bad3f0a None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:57.313777 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58caf976-4b5f-4288-a1cc-4e434bad3f0a None admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:57.314449 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58caf976-4b5f-4288-a1cc-4e434bad3f0a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:42:57.314749 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58caf976-4b5f-4288-a1cc-4e434bad3f0a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:42:57.316199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58caf976-4b5f-4288-a1cc-4e434bad3f0a None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:42:57.327875 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 623/1259] 10.222.0.27 () {56 vars in 1148 bytes} [Sun Sep 11 00:42:57 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:42:57.333821 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b25bcc-dab1-4314-b384-3da2c325b89b None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:42:57.359147 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:42:57.431640 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b25bcc-dab1-4314-b384-3da2c325b89b None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:42:57.464753 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602f2613-1c68-4d6f-9880-09ad66744f85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:42:57.465342 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602f2613-1c68-4d6f-9880-09ad66744f85 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:57.465620 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602f2613-1c68-4d6f-9880-09ad66744f85 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:57.465891 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602f2613-1c68-4d6f-9880-09ad66744f85 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:57.467154 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-602f2613-1c68-4d6f-9880-09ad66744f85 None admin] RBAC: Adding query filter params (role.id=7d9213c1187c4b5182bf8132cf23ea19, scope.project.id=b504a5058bc6474c8cc077dfeec16799, user.id=cdbfc2ccd73e4a5eb8c999be5b094619) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:42:57.467569 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-602f2613-1c68-4d6f-9880-09ad66744f85 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:42:57.468984 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-602f2613-1c68-4d6f-9880-09ad66744f85 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:42:57.474753 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 637/1260] 10.222.0.27 () {56 vars in 1374 bytes} [Sun Sep 11 00:42:57 2022] GET /identity/v3/role_assignments?user.id=cdbfc2ccd73e4a5eb8c999be5b094619&scope.project.id=b504a5058bc6474c8cc077dfeec16799&role.id=7d9213c1187c4b5182bf8132cf23ea19 => generated 259 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:42:59.615760 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd024ab-ddb0-4d08-acc8-002e40175346 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:59.616324 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd024ab-ddb0-4d08-acc8-002e40175346 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:59.616324 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd024ab-ddb0-4d08-acc8-002e40175346 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:59.617639 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 624/1261] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:42:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:42:59.622675 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741d84e8-3a71-4e07-88a0-c5dad8e29006 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:42:59.623402 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741d84e8-3a71-4e07-88a0-c5dad8e29006 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:42:59.623402 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741d84e8-3a71-4e07-88a0-c5dad8e29006 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:42:59.641616 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-741d84e8-3a71-4e07-88a0-c5dad8e29006 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:42:59.642827 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-741d84e8-3a71-4e07-88a0-c5dad8e29006 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:42:59.707616 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 638/1262] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:42:59 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:42:59.973856 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd024ab-ddb0-4d08-acc8-002e40175346 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:42:59.985387 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:43:00.075106 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd024ab-ddb0-4d08-acc8-002e40175346 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:43:00.109618 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b96b2b4-0365-4dc2-a226-e222d4fd0268 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:43:00.110242 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b96b2b4-0365-4dc2-a226-e222d4fd0268 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:43:00.110516 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b96b2b4-0365-4dc2-a226-e222d4fd0268 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:43:00.110799 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b96b2b4-0365-4dc2-a226-e222d4fd0268 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:43:00.120637 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b96b2b4-0365-4dc2-a226-e222d4fd0268 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:43:00.126104 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-8b96b2b4-0365-4dc2-a226-e222d4fd0268 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:43:00.126104 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:43:00.126104 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:43:00.126104 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:43:00.126104 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:43:00.126104 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:43:00.126104 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:43:00.126104 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:43:00.126104 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:43:00.126104 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:43:00.126104 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:43:00.126104 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:43:00.126104 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:43:00.126104 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:43:00.126104 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:43:00.126104 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:43:00.127235 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:43:00.127235 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:43:00.127235 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:43:00.127235 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:43:00.127235 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:43:00.127235 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:43:00.127235 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:43:00.127235 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:43:00.127235 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:43:00.127235 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:43:00.127235 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:43:00.127235 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:43:00.127235 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:43:00.127235 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:43:00.127235 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:43:00.127235 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:43:00.128276 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:43:00.128276 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:43:00.128276 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:43:00.128276 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:43:00.128276 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:43:00.128276 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:43:00.128276 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:43:00.128276 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:43:00.128276 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:43:00.128276 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:43:00.128276 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:43:00.128276 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:43:00.128276 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 625/1263] 10.222.0.27 () {56 vars in 1096 bytes} [Sun Sep 11 00:42:59 2022] GET /identity/v3/roles/admin => generated 83 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:43:00.133262 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-741d84e8-3a71-4e07-88a0-c5dad8e29006 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:43:00.150781 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:43:00.229401 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-741d84e8-3a71-4e07-88a0-c5dad8e29006 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:43:00.263974 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aec13cc-d341-4e65-b0ec-856b304c8398 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:43:00.264689 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aec13cc-d341-4e65-b0ec-856b304c8398 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:43:00.264965 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aec13cc-d341-4e65-b0ec-856b304c8398 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:43:00.265233 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aec13cc-d341-4e65-b0ec-856b304c8398 None admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:43:00.265883 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aec13cc-d341-4e65-b0ec-856b304c8398 None admin] RBAC: Adding query filter params (name=admin) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:43:00.266177 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aec13cc-d341-4e65-b0ec-856b304c8398 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:43:00.267551 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aec13cc-d341-4e65-b0ec-856b304c8398 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:43:00.275261 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 639/1264] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:43:00 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:43:00.281191 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b96b2b4-0365-4dc2-a226-e222d4fd0268 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:43:00.346296 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:43:00.378926 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b96b2b4-0365-4dc2-a226-e222d4fd0268 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:43:00.411660 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb3d028-7502-4bdc-ac11-74518d204ec4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:43:00.412265 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb3d028-7502-4bdc-ac11-74518d204ec4 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:43:00.412539 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb3d028-7502-4bdc-ac11-74518d204ec4 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:43:00.412809 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb3d028-7502-4bdc-ac11-74518d204ec4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:43:00.422920 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cb3d028-7502-4bdc-ac11-74518d204ec4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:43:00.424349 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cb3d028-7502-4bdc-ac11-74518d204ec4 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:43:00.438317 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-5cb3d028-7502-4bdc-ac11-74518d204ec4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:43:00.438317 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:43:00.438317 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:43:00.438317 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:43:00.438317 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:43:00.438317 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:43:00.438317 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:43:00.438317 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:43:00.438317 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:43:00.438317 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:43:00.438317 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:43:00.438317 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:43:00.438317 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:43:00.438317 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:43:00.438317 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:43:00.438317 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:43:00.443583 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:43:00.443583 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:43:00.443583 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:43:00.443583 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:43:00.443583 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:43:00.443583 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:43:00.443583 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:43:00.443583 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:43:00.443583 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:43:00.443583 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:43:00.443583 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:43:00.443583 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:43:00.443583 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:43:00.443583 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:43:00.443583 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:43:00.443583 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:43:00.443583 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:43:00.444673 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:43:00.444673 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:43:00.444673 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:43:00.444673 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 626/1265] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:43:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:43:00.445296 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aec13cc-d341-4e65-b0ec-856b304c8398 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:43:00.538239 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:43:00.560631 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aec13cc-d341-4e65-b0ec-856b304c8398 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:43:00.593273 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fca8f53-b177-4e36-bebe-f497c8c49a41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:43:00.593875 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fca8f53-b177-4e36-bebe-f497c8c49a41 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:43:00.594155 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fca8f53-b177-4e36-bebe-f497c8c49a41 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:43:00.594426 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fca8f53-b177-4e36-bebe-f497c8c49a41 None admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:43:00.595095 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fca8f53-b177-4e36-bebe-f497c8c49a41 None admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:43:00.595396 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fca8f53-b177-4e36-bebe-f497c8c49a41 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:43:00.596756 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fca8f53-b177-4e36-bebe-f497c8c49a41 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:43:00.608720 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 640/1266] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:43:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:43:00.614580 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb3d028-7502-4bdc-ac11-74518d204ec4 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:43:00.711919 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb3d028-7502-4bdc-ac11-74518d204ec4 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:43:00.731299 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:43:00.745763 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021f78f3-88e2-49b5-947b-d567ad24145d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:43:00.746376 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021f78f3-88e2-49b5-947b-d567ad24145d None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:43:00.746658 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021f78f3-88e2-49b5-947b-d567ad24145d None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:43:00.746956 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021f78f3-88e2-49b5-947b-d567ad24145d None admin] PATH_INFO: `/v3/users/placement` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:43:00.755116 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-021f78f3-88e2-49b5-947b-d567ad24145d None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:43:00.756555 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-021f78f3-88e2-49b5-947b-d567ad24145d None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:43:00.767531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-021f78f3-88e2-49b5-947b-d567ad24145d None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 11 00:43:00.767531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:43:00.767531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:43:00.767531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:43:00.767531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:43:00.767531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:43:00.767531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:43:00.767531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:43:00.767531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:43:00.767531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:43:00.767531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:43:00.767531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:43:00.767531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 11 00:43:00.767531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 11 00:43:00.767531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 11 00:43:00.767531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 11 00:43:00.768655 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:43:00.768655 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:43:00.768655 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 11 00:43:00.768655 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:43:00.768655 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 11 00:43:00.768655 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:43:00.768655 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:43:00.768655 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:43:00.768655 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:43:00.768655 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:43:00.768655 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:43:00.768655 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:43:00.768655 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:43:00.768655 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:43:00.768655 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:43:00.768655 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:43:00.768655 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:43:00.768655 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:43:00.769852 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:43:00.769852 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:43:00.769852 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 11 00:43:00.769852 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 11 00:43:00.769852 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 11 00:43:00.769852 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 11 00:43:00.769852 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 11 00:43:00.769852 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 11 00:43:00.769852 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 11 00:43:00.769852 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:43:00.769852 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 627/1267] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:43:00 2022] GET /identity/v3/users/placement => generated 87 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:43:00.774562 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fca8f53-b177-4e36-bebe-f497c8c49a41 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:43:00.872379 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fca8f53-b177-4e36-bebe-f497c8c49a41 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:43:00.905577 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d934fd-a77b-4d9a-9769-be819e22e222 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:43:00.906161 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d934fd-a77b-4d9a-9769-be819e22e222 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:43:00.906441 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d934fd-a77b-4d9a-9769-be819e22e222 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:43:00.906714 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d934fd-a77b-4d9a-9769-be819e22e222 None admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:43:00.907461 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1d934fd-a77b-4d9a-9769-be819e22e222 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:43:00.907765 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1d934fd-a77b-4d9a-9769-be819e22e222 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:43:00.909451 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1d934fd-a77b-4d9a-9769-be819e22e222 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:43:00.917746 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:43:00.927104 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 641/1268] 10.222.0.27 () {56 vars in 1143 bytes} [Sun Sep 11 00:43:00 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:43:00.933005 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021f78f3-88e2-49b5-947b-d567ad24145d None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:43:01.030382 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021f78f3-88e2-49b5-947b-d567ad24145d None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:43:01.062469 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be074c88-c30f-4fc9-858f-e51d5b74c9f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:43:01.063100 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be074c88-c30f-4fc9-858f-e51d5b74c9f5 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:43:01.063374 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be074c88-c30f-4fc9-858f-e51d5b74c9f5 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:43:01.063638 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be074c88-c30f-4fc9-858f-e51d5b74c9f5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:43:01.073660 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be074c88-c30f-4fc9-858f-e51d5b74c9f5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:43:01.075172 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be074c88-c30f-4fc9-858f-e51d5b74c9f5 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:43:01.089445 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-be074c88-c30f-4fc9-858f-e51d5b74c9f5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:43:01.089445 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:43:01.089445 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:43:01.089445 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:43:01.089445 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:43:01.089445 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:43:01.089445 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:43:01.089445 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:43:01.089445 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:43:01.089445 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:43:01.089445 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:43:01.089445 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:43:01.089445 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:43:01.089445 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:43:01.089445 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:43:01.089445 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:43:01.090625 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:43:01.090625 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:43:01.090625 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:43:01.090625 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:43:01.090625 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:43:01.090625 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:43:01.090625 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:43:01.090625 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:43:01.090625 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:43:01.090625 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:43:01.090625 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:43:01.090625 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:43:01.090625 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:43:01.090625 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:43:01.090625 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:43:01.090625 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:43:01.090625 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:43:01.092664 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:43:01.092664 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:43:01.092664 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:43:01.092664 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 628/1269] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:43:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:43:01.096815 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d934fd-a77b-4d9a-9769-be819e22e222 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:43:01.193305 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d934fd-a77b-4d9a-9769-be819e22e222 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:43:01.226053 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c411eb0-0c94-4e7f-a26c-40802ac51962 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:43:01.226643 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c411eb0-0c94-4e7f-a26c-40802ac51962 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:43:01.226948 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c411eb0-0c94-4e7f-a26c-40802ac51962 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:43:01.227221 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c411eb0-0c94-4e7f-a26c-40802ac51962 None admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:43:01.227847 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c411eb0-0c94-4e7f-a26c-40802ac51962 None admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:43:01.228151 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c411eb0-0c94-4e7f-a26c-40802ac51962 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:43:01.229505 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c411eb0-0c94-4e7f-a26c-40802ac51962 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:43:01.240411 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 642/1270] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:43:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:43:01.246021 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be074c88-c30f-4fc9-858f-e51d5b74c9f5 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:43:01.277956 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:43:01.345838 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be074c88-c30f-4fc9-858f-e51d5b74c9f5 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:43:01.378136 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1fdc1f2-5f3e-48de-aa36-8719eb1c518a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:43:01.378742 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1fdc1f2-5f3e-48de-aa36-8719eb1c518a None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:43:01.379050 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1fdc1f2-5f3e-48de-aa36-8719eb1c518a None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:43:01.379319 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1fdc1f2-5f3e-48de-aa36-8719eb1c518a None admin] PATH_INFO: `/v3/projects/service` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:43:01.385726 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1fdc1f2-5f3e-48de-aa36-8719eb1c518a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:43:01.387207 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1fdc1f2-5f3e-48de-aa36-8719eb1c518a None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:43:01.398635 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-d1fdc1f2-5f3e-48de-aa36-8719eb1c518a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:43:01.398635 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:43:01.398635 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:43:01.398635 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:43:01.398635 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:43:01.398635 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:43:01.398635 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:43:01.398635 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:43:01.398635 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:43:01.398635 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:43:01.398635 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:43:01.398635 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:43:01.398635 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 11 00:43:01.398635 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 11 00:43:01.398635 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 11 00:43:01.399708 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 11 00:43:01.399708 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:43:01.399708 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:43:01.399708 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:43:01.399708 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:43:01.399708 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:43:01.399708 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:43:01.399708 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:43:01.399708 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:43:01.399708 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:43:01.399708 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:43:01.399708 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:43:01.399708 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:43:01.399708 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:43:01.399708 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:43:01.399708 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:43:01.399708 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:43:01.400807 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 11 00:43:01.400807 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 11 00:43:01.400807 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 11 00:43:01.400807 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 11 00:43:01.400807 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 11 00:43:01.400807 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 11 00:43:01.400807 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:43:01.400807 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:43:01.400807 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 629/1271] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:43:01 2022] GET /identity/v3/projects/service => generated 88 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:43:01.405491 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c411eb0-0c94-4e7f-a26c-40802ac51962 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:43:01.461666 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:43:01.508311 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c411eb0-0c94-4e7f-a26c-40802ac51962 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:43:01.541504 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aa6805f-2936-4bc2-a7cc-d223ee603387 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:43:01.542098 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa6805f-2936-4bc2-a7cc-d223ee603387 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:43:01.542373 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa6805f-2936-4bc2-a7cc-d223ee603387 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:43:01.542641 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa6805f-2936-4bc2-a7cc-d223ee603387 None admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:43:01.543346 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aa6805f-2936-4bc2-a7cc-d223ee603387 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:43:01.543649 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aa6805f-2936-4bc2-a7cc-d223ee603387 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:43:01.545743 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aa6805f-2936-4bc2-a7cc-d223ee603387 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:43:01.557133 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 643/1272] 10.222.0.27 () {56 vars in 1148 bytes} [Sun Sep 11 00:43:01 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:43:01.563888 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1fdc1f2-5f3e-48de-aa36-8719eb1c518a None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:43:01.651850 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:43:01.660760 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1fdc1f2-5f3e-48de-aa36-8719eb1c518a None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:43:01.693605 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90679326-bb31-4009-9681-e54b30a24417 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:43:01.694243 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90679326-bb31-4009-9681-e54b30a24417 None admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:43:01.694520 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90679326-bb31-4009-9681-e54b30a24417 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:43:01.694819 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90679326-bb31-4009-9681-e54b30a24417 None admin] PATH_INFO: `/v3/projects/b504a5058bc6474c8cc077dfeec16799/users/cdbfc2ccd73e4a5eb8c999be5b094619/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:43:01.696674 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90679326-bb31-4009-9681-e54b30a24417 None admin] RBAC: Authorizing `identity:create_grant(project_id=b504a5058bc6474c8cc077dfeec16799, user_id=cdbfc2ccd73e4a5eb8c999be5b094619, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:43:01.698105 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90679326-bb31-4009-9681-e54b30a24417 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:43:01.710227 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 630/1273] 10.222.0.27 () {58 vars in 1439 bytes} [Sun Sep 11 00:43:01 2022] PUT /identity/v3/projects/b504a5058bc6474c8cc077dfeec16799/users/cdbfc2ccd73e4a5eb8c999be5b094619/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:43:03.842936 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f610fae7-308c-4bbd-a89b-c18eda3bf05a None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:43:03.843480 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f610fae7-308c-4bbd-a89b-c18eda3bf05a None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:43:03.843630 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f610fae7-308c-4bbd-a89b-c18eda3bf05a None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:43:03.845044 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 644/1274] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:43:03 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:43:03.850258 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c781f9f-4188-4071-b79a-6245bf02cef8 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:43:03.850663 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c781f9f-4188-4071-b79a-6245bf02cef8 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:43:03.850883 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c781f9f-4188-4071-b79a-6245bf02cef8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:43:03.869582 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-6c781f9f-4188-4071-b79a-6245bf02cef8 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:43:03.870743 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-6c781f9f-4188-4071-b79a-6245bf02cef8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:43:03.935450 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 631/1275] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:43:03 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:43:04.202189 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f610fae7-308c-4bbd-a89b-c18eda3bf05a None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:43:04.300434 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f610fae7-308c-4bbd-a89b-c18eda3bf05a None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:43:04.316804 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:43:04.333116 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-484f2d75-ffce-4aa1-80ff-717c57445db6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:43:04.333739 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484f2d75-ffce-4aa1-80ff-717c57445db6 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:43:04.334015 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484f2d75-ffce-4aa1-80ff-717c57445db6 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:43:04.334282 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484f2d75-ffce-4aa1-80ff-717c57445db6 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:43:04.344580 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-484f2d75-ffce-4aa1-80ff-717c57445db6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 11 00:43:04.349375 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-484f2d75-ffce-4aa1-80ff-717c57445db6 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:43:04.349375 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:43:04.349375 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:43:04.349375 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:43:04.349375 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:43:04.349375 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:43:04.349375 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:43:04.349375 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:43:04.349375 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:43:04.349375 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:43:04.349375 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:43:04.349375 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:43:04.349375 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 11 00:43:04.349375 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 11 00:43:04.349375 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 11 00:43:04.349375 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 11 00:43:04.350470 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 11 00:43:04.350470 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 11 00:43:04.350470 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 11 00:43:04.350470 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 11 00:43:04.350470 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:43:04.350470 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:43:04.350470 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:43:04.350470 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:43:04.350470 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:43:04.350470 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:43:04.350470 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:43:04.350470 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:43:04.350470 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:43:04.350470 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:43:04.350470 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:43:04.350470 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:43:04.351535 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:43:04.351535 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:43:04.351535 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:43:04.351535 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:43:04.351535 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 11 00:43:04.351535 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 11 00:43:04.351535 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 11 00:43:04.351535 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 11 00:43:04.351535 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 11 00:43:04.351535 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 11 00:43:04.351535 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 11 00:43:04.351535 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:43:04.351535 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 645/1276] 10.222.0.27 () {56 vars in 1096 bytes} [Sun Sep 11 00:43:04 2022] GET /identity/v3/roles/admin => generated 83 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:43:04.356449 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c781f9f-4188-4071-b79a-6245bf02cef8 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:43:04.435189 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:43:04.453285 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c781f9f-4188-4071-b79a-6245bf02cef8 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:43:04.491395 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69fea805-5fb8-4189-a6a6-dae5a96d2e61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:43:04.492116 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69fea805-5fb8-4189-a6a6-dae5a96d2e61 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:43:04.492531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69fea805-5fb8-4189-a6a6-dae5a96d2e61 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:43:04.492810 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69fea805-5fb8-4189-a6a6-dae5a96d2e61 None admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:43:04.493612 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69fea805-5fb8-4189-a6a6-dae5a96d2e61 None admin] RBAC: Adding query filter params (name=admin) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:43:04.493913 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69fea805-5fb8-4189-a6a6-dae5a96d2e61 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:43:04.495705 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69fea805-5fb8-4189-a6a6-dae5a96d2e61 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:43:04.505905 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 632/1277] 10.222.0.27 () {56 vars in 1099 bytes} [Sun Sep 11 00:43:04 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:43:04.512140 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-484f2d75-ffce-4aa1-80ff-717c57445db6 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:43:04.612779 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-484f2d75-ffce-4aa1-80ff-717c57445db6 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:43:04.645324 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1955d32-0331-42c8-a12e-e3083e18991c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:43:04.645946 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1955d32-0331-42c8-a12e-e3083e18991c None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:43:04.646223 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1955d32-0331-42c8-a12e-e3083e18991c None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:43:04.646496 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1955d32-0331-42c8-a12e-e3083e18991c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:43:04.656767 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1955d32-0331-42c8-a12e-e3083e18991c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:43:04.658476 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1955d32-0331-42c8-a12e-e3083e18991c None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:43:04.672620 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-b1955d32-0331-42c8-a12e-e3083e18991c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:43:04.672620 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:43:04.672620 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:43:04.672620 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:43:04.672620 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:43:04.672620 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:43:04.672620 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:43:04.672620 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:43:04.672620 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:43:04.672620 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:43:04.672620 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:43:04.672620 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:43:04.672620 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:43:04.672620 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:43:04.672620 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:43:04.672620 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:43:04.678485 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:43:04.678485 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:43:04.678485 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:43:04.678485 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:43:04.678485 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:43:04.678485 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:43:04.678485 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:43:04.678485 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:43:04.678485 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:43:04.678485 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:43:04.678485 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:43:04.678485 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:43:04.678485 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:43:04.678485 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:43:04.678485 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:43:04.678485 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:43:04.678485 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:43:04.679657 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:43:04.679657 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:43:04.679657 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:43:04.679657 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 646/1278] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:43:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:43:04.679945 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69fea805-5fb8-4189-a6a6-dae5a96d2e61 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:43:04.776858 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69fea805-5fb8-4189-a6a6-dae5a96d2e61 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:43:04.810984 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641ac6ee-31ae-4c53-96c0-7d7226eec59e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:43:04.811601 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641ac6ee-31ae-4c53-96c0-7d7226eec59e None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:43:04.811881 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641ac6ee-31ae-4c53-96c0-7d7226eec59e None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:43:04.812155 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641ac6ee-31ae-4c53-96c0-7d7226eec59e None admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:43:04.812790 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-641ac6ee-31ae-4c53-96c0-7d7226eec59e None admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:43:04.813092 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-641ac6ee-31ae-4c53-96c0-7d7226eec59e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:43:04.814463 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-641ac6ee-31ae-4c53-96c0-7d7226eec59e None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:43:04.826330 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 633/1279] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:43:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:43:04.832336 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1955d32-0331-42c8-a12e-e3083e18991c None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:43:04.864771 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:43:04.934054 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1955d32-0331-42c8-a12e-e3083e18991c None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:43:04.965796 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6dc64e0-309a-4e1b-a454-706229a77169 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:43:04.967173 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6dc64e0-309a-4e1b-a454-706229a77169 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:43:04.967619 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6dc64e0-309a-4e1b-a454-706229a77169 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:43:04.967888 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6dc64e0-309a-4e1b-a454-706229a77169 None admin] PATH_INFO: `/v3/users/placement` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:43:04.975034 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6dc64e0-309a-4e1b-a454-706229a77169 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:43:04.976451 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6dc64e0-309a-4e1b-a454-706229a77169 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:43:04.988182 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-a6dc64e0-309a-4e1b-a454-706229a77169 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 11 00:43:04.988182 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:43:04.988182 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:43:04.988182 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:43:04.988182 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:43:04.988182 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:43:04.988182 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:43:04.988182 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:43:04.988182 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:43:04.988182 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:43:04.988182 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:43:04.988182 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:43:04.988182 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 11 00:43:04.988182 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 11 00:43:04.988182 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 11 00:43:04.988182 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 11 00:43:04.989296 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:43:04.989296 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:43:04.989296 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 11 00:43:04.989296 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:43:04.989296 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 11 00:43:04.989296 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 11 00:43:04.989296 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:43:04.989296 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:43:04.989296 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:43:04.989296 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:43:04.989296 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:43:04.989296 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:43:04.989296 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:43:04.989296 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:43:04.989296 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:43:04.989296 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:43:04.989296 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:43:04.989296 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:43:04.990649 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:43:04.990649 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:43:04.990649 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 11 00:43:04.990649 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 11 00:43:04.990649 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 11 00:43:04.990649 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 11 00:43:04.990649 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 11 00:43:04.990649 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 11 00:43:04.990649 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 11 00:43:04.990649 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:43:04.990649 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 647/1280] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:43:04 2022] GET /identity/v3/users/placement => generated 87 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:43:04.995045 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641ac6ee-31ae-4c53-96c0-7d7226eec59e None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:43:04.999711 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:43:05.094942 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641ac6ee-31ae-4c53-96c0-7d7226eec59e None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:43:05.128768 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7e82772-6226-4ad6-953f-1f542e08c1b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:43:05.129307 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e82772-6226-4ad6-953f-1f542e08c1b0 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:43:05.129586 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e82772-6226-4ad6-953f-1f542e08c1b0 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:43:05.129859 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e82772-6226-4ad6-953f-1f542e08c1b0 None admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:43:05.130580 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7e82772-6226-4ad6-953f-1f542e08c1b0 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:43:05.130909 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7e82772-6226-4ad6-953f-1f542e08c1b0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:43:05.132343 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7e82772-6226-4ad6-953f-1f542e08c1b0 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:43:05.148993 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 634/1281] 10.222.0.27 () {56 vars in 1143 bytes} [Sun Sep 11 00:43:04 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:43:05.155330 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6dc64e0-309a-4e1b-a454-706229a77169 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:43:05.241699 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:43:05.252322 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6dc64e0-309a-4e1b-a454-706229a77169 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:43:05.285367 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c863aa79-8665-449c-b1f5-6a8c6254aac6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:43:05.285963 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c863aa79-8665-449c-b1f5-6a8c6254aac6 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:43:05.286241 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c863aa79-8665-449c-b1f5-6a8c6254aac6 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:43:05.286512 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c863aa79-8665-449c-b1f5-6a8c6254aac6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:43:05.351168 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c863aa79-8665-449c-b1f5-6a8c6254aac6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:43:05.352609 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c863aa79-8665-449c-b1f5-6a8c6254aac6 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:43:05.437694 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:43:05.447694 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 648/1282] 10.222.0.27 () {58 vars in 1282 bytes} [Sun Sep 11 00:43:05 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 11 00:43:05.453619 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7e82772-6226-4ad6-953f-1f542e08c1b0 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:43:05.518709 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:43:05.556634 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7e82772-6226-4ad6-953f-1f542e08c1b0 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:43:05.589680 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c51de17-68b7-45a6-ab9f-6c8becae9705 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:43:05.590304 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c51de17-68b7-45a6-ab9f-6c8becae9705 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:43:05.590582 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c51de17-68b7-45a6-ab9f-6c8becae9705 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:43:05.590881 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c51de17-68b7-45a6-ab9f-6c8becae9705 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:43:05.600787 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c51de17-68b7-45a6-ab9f-6c8becae9705 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:43:05.602223 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c51de17-68b7-45a6-ab9f-6c8becae9705 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:43:05.616978 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-9c51de17-68b7-45a6-ab9f-6c8becae9705 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:43:05.616978 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:43:05.616978 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:43:05.616978 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:43:05.616978 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:43:05.616978 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:43:05.616978 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:43:05.616978 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:43:05.616978 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:43:05.616978 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:43:05.616978 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:43:05.616978 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:43:05.616978 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 11 00:43:05.616978 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 11 00:43:05.616978 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 11 00:43:05.616978 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 11 00:43:05.618271 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:43:05.618271 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:43:05.618271 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:43:05.618271 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:43:05.618271 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:43:05.618271 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:43:05.618271 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:43:05.618271 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:43:05.618271 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:43:05.618271 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:43:05.618271 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:43:05.618271 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:43:05.618271 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:43:05.618271 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:43:05.618271 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:43:05.618271 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:43:05.618271 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 11 00:43:05.619513 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 11 00:43:05.619513 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 11 00:43:05.619513 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:43:05.619513 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 635/1283] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:43:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:43:05.624391 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c863aa79-8665-449c-b1f5-6a8c6254aac6 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:43:05.720137 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c863aa79-8665-449c-b1f5-6a8c6254aac6 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:43:05.752891 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f73033-be3d-4195-9967-4e1bce93bb28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:43:05.753485 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f73033-be3d-4195-9967-4e1bce93bb28 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:43:05.753763 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f73033-be3d-4195-9967-4e1bce93bb28 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:43:05.754035 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f73033-be3d-4195-9967-4e1bce93bb28 None admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:43:05.754665 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31f73033-be3d-4195-9967-4e1bce93bb28 None admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:43:05.754994 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31f73033-be3d-4195-9967-4e1bce93bb28 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:43:05.756346 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31f73033-be3d-4195-9967-4e1bce93bb28 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:43:05.767573 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 649/1284] 10.222.0.27 () {56 vars in 1109 bytes} [Sun Sep 11 00:43:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:43:05.773041 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c51de17-68b7-45a6-ab9f-6c8becae9705 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:43:05.870190 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c51de17-68b7-45a6-ab9f-6c8becae9705 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:43:05.883829 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:43:05.904345 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16f19946-5bff-4b14-92c0-a5e3e7138f43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:43:05.904956 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f19946-5bff-4b14-92c0-a5e3e7138f43 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:43:05.905234 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f19946-5bff-4b14-92c0-a5e3e7138f43 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:43:05.905503 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f19946-5bff-4b14-92c0-a5e3e7138f43 None admin] PATH_INFO: `/v3/projects/service` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:43:05.912232 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16f19946-5bff-4b14-92c0-a5e3e7138f43 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:43:05.913667 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16f19946-5bff-4b14-92c0-a5e3e7138f43 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:43:05.926185 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application [None req-16f19946-5bff-4b14-92c0-a5e3e7138f43 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:43:05.926185 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:43:05.926185 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:43:05.926185 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:43:05.926185 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:43:05.926185 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:43:05.926185 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:43:05.926185 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:43:05.926185 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:43:05.926185 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:43:05.926185 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:43:05.926185 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:43:05.926185 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 11 00:43:05.926185 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 11 00:43:05.926185 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 11 00:43:05.927248 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 11 00:43:05.927248 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:43:05.927248 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:43:05.927248 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:43:05.927248 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:43:05.927248 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:43:05.927248 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:43:05.927248 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:43:05.927248 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application with Lock( Sep 11 00:43:05.927248 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:43:05.927248 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:43:05.927248 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:43:05.927248 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:43:05.927248 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:43:05.927248 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:43:05.927248 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:43:05.927248 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:43:05.928551 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 11 00:43:05.928551 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 11 00:43:05.928551 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 11 00:43:05.928551 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 11 00:43:05.928551 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 11 00:43:05.928551 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 11 00:43:05.928551 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 11 00:43:05.928551 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: ERROR keystone.server.flask.application Sep 11 00:43:05.928551 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 636/1285] 10.222.0.27 () {56 vars in 1111 bytes} [Sun Sep 11 00:43:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:43:05.933113 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f73033-be3d-4195-9967-4e1bce93bb28 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:43:05.981103 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:43:06.029809 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f73033-be3d-4195-9967-4e1bce93bb28 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:43:06.063005 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f31162b-9912-4425-884f-4e58a0597b1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:43:06.063627 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f31162b-9912-4425-884f-4e58a0597b1a None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:43:06.063905 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f31162b-9912-4425-884f-4e58a0597b1a None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:43:06.064175 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f31162b-9912-4425-884f-4e58a0597b1a None admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:43:06.064858 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f31162b-9912-4425-884f-4e58a0597b1a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:43:06.065161 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f31162b-9912-4425-884f-4e58a0597b1a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:43:06.066609 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f31162b-9912-4425-884f-4e58a0597b1a None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:43:06.079753 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 650/1286] 10.222.0.27 () {56 vars in 1148 bytes} [Sun Sep 11 00:43:05 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:43:06.085720 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16f19946-5bff-4b14-92c0-a5e3e7138f43 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:43:06.181992 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16f19946-5bff-4b14-92c0-a5e3e7138f43 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:43:06.214303 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f1e2dc-b196-4dcc-b3e0-4f6d964d6004 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:43:06.214920 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f1e2dc-b196-4dcc-b3e0-4f6d964d6004 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:43:06.215197 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f1e2dc-b196-4dcc-b3e0-4f6d964d6004 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:43:06.215464 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f1e2dc-b196-4dcc-b3e0-4f6d964d6004 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:43:06.216737 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54f1e2dc-b196-4dcc-b3e0-4f6d964d6004 None admin] RBAC: Adding query filter params (role.id=7d9213c1187c4b5182bf8132cf23ea19, scope.project.id=b504a5058bc6474c8cc077dfeec16799, user.id=cdbfc2ccd73e4a5eb8c999be5b094619) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:43:06.217112 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54f1e2dc-b196-4dcc-b3e0-4f6d964d6004 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:43:06.218529 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54f1e2dc-b196-4dcc-b3e0-4f6d964d6004 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:43:06.224130 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 637/1287] 10.222.0.27 () {56 vars in 1374 bytes} [Sun Sep 11 00:43:06 2022] GET /identity/v3/role_assignments?user.id=cdbfc2ccd73e4a5eb8c999be5b094619&scope.project.id=b504a5058bc6474c8cc077dfeec16799&role.id=7d9213c1187c4b5182bf8132cf23ea19 => generated 608 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:43:08.379341 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9289aaf4-8957-480a-9106-1fa4053c9eee None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:43:08.379875 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9289aaf4-8957-480a-9106-1fa4053c9eee None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:43:08.379875 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9289aaf4-8957-480a-9106-1fa4053c9eee None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:43:08.381118 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 651/1288] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:43:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:43:08.386208 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3ce5295-4f2b-4a4e-b022-5acb0f3f5bbf None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:43:08.386617 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3ce5295-4f2b-4a4e-b022-5acb0f3f5bbf None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:43:08.386736 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3ce5295-4f2b-4a4e-b022-5acb0f3f5bbf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:43:08.405632 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-b3ce5295-4f2b-4a4e-b022-5acb0f3f5bbf None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:43:08.406815 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-b3ce5295-4f2b-4a4e-b022-5acb0f3f5bbf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:43:08.435406 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:43:08.472249 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 638/1289] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:43:08 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:43:08.748570 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9289aaf4-8957-480a-9106-1fa4053c9eee None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:43:08.846973 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9289aaf4-8957-480a-9106-1fa4053c9eee None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:43:08.857164 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:43:08.880218 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0996d852-04ee-4754-b532-5b130ed66fba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:43:08.880833 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0996d852-04ee-4754-b532-5b130ed66fba None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:43:08.881110 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0996d852-04ee-4754-b532-5b130ed66fba None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:43:08.881380 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0996d852-04ee-4754-b532-5b130ed66fba None admin] PATH_INFO: `/v3/services/placement` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:43:08.881957 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0996d852-04ee-4754-b532-5b130ed66fba None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:43:08.883343 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0996d852-04ee-4754-b532-5b130ed66fba None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:43:08.892594 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-0996d852-04ee-4754-b532-5b130ed66fba None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 11 00:43:08.892594 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:43:08.892594 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 11 00:43:08.892594 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 11 00:43:08.892594 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 11 00:43:08.892594 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 11 00:43:08.892594 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 11 00:43:08.892594 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 11 00:43:08.892594 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 11 00:43:08.892594 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:43:08.892594 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 11 00:43:08.892594 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:43:08.892594 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:43:08.892594 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:43:08.892594 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:43:08.892594 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:43:08.892594 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:43:08.893839 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:43:08.893839 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:43:08.893839 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:43:08.893839 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:43:08.893839 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:43:08.893839 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:43:08.893839 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 11 00:43:08.893839 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 11 00:43:08.893839 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 11 00:43:08.893839 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 11 00:43:08.893839 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:43:08.893839 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:43:08.893839 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:43:08.893839 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:43:08.893839 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:43:08.893839 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:43:08.894975 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:43:08.894975 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:43:08.894975 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:43:08.894975 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:43:08.894975 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:43:08.894975 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:43:08.894975 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:43:08.894975 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:43:08.894975 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:43:08.894975 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:43:08.894975 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 11 00:43:08.894975 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 11 00:43:08.894975 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 11 00:43:08.894975 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:43:08.894975 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 652/1290] 10.222.0.27 () {56 vars in 1117 bytes} [Sun Sep 11 00:43:08 2022] GET /identity/v3/services/placement => generated 90 bytes in 146 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:43:08.899543 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3ce5295-4f2b-4a4e-b022-5acb0f3f5bbf None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:43:08.994885 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3ce5295-4f2b-4a4e-b022-5acb0f3f5bbf None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:43:09.030504 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361a9d14-6652-419e-af1f-5361ff54eac7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:43:09.031060 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361a9d14-6652-419e-af1f-5361ff54eac7 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:43:09.031333 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361a9d14-6652-419e-af1f-5361ff54eac7 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:43:09.031599 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361a9d14-6652-419e-af1f-5361ff54eac7 None admin] PATH_INFO: `/v3/services` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:43:09.032217 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361a9d14-6652-419e-af1f-5361ff54eac7 None admin] RBAC: Adding query filter params (name=placement) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:43:09.032515 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361a9d14-6652-419e-af1f-5361ff54eac7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:43:09.034168 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361a9d14-6652-419e-af1f-5361ff54eac7 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:43:09.038759 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 639/1291] 10.222.0.27 () {56 vars in 1116 bytes} [Sun Sep 11 00:43:08 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:43:09.045001 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0996d852-04ee-4754-b532-5b130ed66fba None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:43:09.142865 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0996d852-04ee-4754-b532-5b130ed66fba None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:43:09.175362 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb4643b-7644-4b18-a90d-bf27d4a414cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:43:09.175965 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb4643b-7644-4b18-a90d-bf27d4a414cb None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:43:09.176246 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb4643b-7644-4b18-a90d-bf27d4a414cb None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:43:09.176517 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb4643b-7644-4b18-a90d-bf27d4a414cb None admin] PATH_INFO: `/v3/services` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:43:09.177144 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceb4643b-7644-4b18-a90d-bf27d4a414cb None admin] RBAC: Adding query filter params (type=placement) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:43:09.177447 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceb4643b-7644-4b18-a90d-bf27d4a414cb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:43:09.178828 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceb4643b-7644-4b18-a90d-bf27d4a414cb None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:43:09.183118 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 653/1292] 10.222.0.27 () {56 vars in 1116 bytes} [Sun Sep 11 00:43:09 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:43:11.309427 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f756dccc-deb5-48f7-b9e4-9ed592f5a98e None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:43:11.310139 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f756dccc-deb5-48f7-b9e4-9ed592f5a98e None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:43:11.310139 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f756dccc-deb5-48f7-b9e4-9ed592f5a98e None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:43:11.311428 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 640/1293] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:43:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:43:11.316808 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a93f5d-8ca1-4e71-a015-483ac2e1ef09 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:43:11.317229 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a93f5d-8ca1-4e71-a015-483ac2e1ef09 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:43:11.317357 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a93f5d-8ca1-4e71-a015-483ac2e1ef09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:43:11.335682 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-a6a93f5d-8ca1-4e71-a015-483ac2e1ef09 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:43:11.336861 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-a6a93f5d-8ca1-4e71-a015-483ac2e1ef09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:43:11.365831 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:43:11.402280 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 654/1294] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:43:11 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:43:11.674487 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f756dccc-deb5-48f7-b9e4-9ed592f5a98e None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:43:11.705892 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:43:11.775301 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f756dccc-deb5-48f7-b9e4-9ed592f5a98e None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:43:11.808334 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77e0a187-0361-40c9-9a69-7c5f9dccce28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:43:11.808929 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e0a187-0361-40c9-9a69-7c5f9dccce28 None admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:43:11.809210 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e0a187-0361-40c9-9a69-7c5f9dccce28 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:43:11.809992 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e0a187-0361-40c9-9a69-7c5f9dccce28 None admin] PATH_INFO: `/v3/services` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:43:11.810981 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77e0a187-0361-40c9-9a69-7c5f9dccce28 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:43:11.812350 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77e0a187-0361-40c9-9a69-7c5f9dccce28 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:43:11.826111 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 641/1295] 10.222.0.27 () {60 vars in 1141 bytes} [Sun Sep 11 00:43:11 2022] POST /identity/v3/services => generated 246 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:43:13.981862 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2eb7c2d-ef4a-445e-a252-7bba36e6b2f6 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:43:13.982388 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2eb7c2d-ef4a-445e-a252-7bba36e6b2f6 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:43:13.982388 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2eb7c2d-ef4a-445e-a252-7bba36e6b2f6 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:43:13.984412 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 655/1296] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:43:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:43:13.989541 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b5893a1-0971-4d51-ad0e-32cc74e134ea None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:43:13.989923 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b5893a1-0971-4d51-ad0e-32cc74e134ea None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:43:13.990097 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b5893a1-0971-4d51-ad0e-32cc74e134ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:43:14.007937 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-1b5893a1-0971-4d51-ad0e-32cc74e134ea None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:43:14.009114 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-1b5893a1-0971-4d51-ad0e-32cc74e134ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:43:14.080541 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 642/1297] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:43:13 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:43:14.355746 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2eb7c2d-ef4a-445e-a252-7bba36e6b2f6 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:43:14.454910 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2eb7c2d-ef4a-445e-a252-7bba36e6b2f6 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:43:14.491498 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f51d3e12-b14b-4794-935c-bdff182fc9e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:43:14.492177 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f51d3e12-b14b-4794-935c-bdff182fc9e4 None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:43:14.492571 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f51d3e12-b14b-4794-935c-bdff182fc9e4 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:43:14.492865 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f51d3e12-b14b-4794-935c-bdff182fc9e4 None admin] PATH_INFO: `/v3/services/placement` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:43:14.493590 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f51d3e12-b14b-4794-935c-bdff182fc9e4 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:43:14.495458 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f51d3e12-b14b-4794-935c-bdff182fc9e4 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:43:14.507444 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-f51d3e12-b14b-4794-935c-bdff182fc9e4 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 11 00:43:14.507444 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:43:14.507444 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 11 00:43:14.507444 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 11 00:43:14.507444 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 11 00:43:14.507444 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 11 00:43:14.507444 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 11 00:43:14.507444 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 11 00:43:14.507444 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 11 00:43:14.507444 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:43:14.507444 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 11 00:43:14.507444 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:43:14.507444 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:43:14.507444 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:43:14.507444 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:43:14.507444 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:43:14.507444 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:43:14.508637 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:43:14.508637 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:43:14.508637 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:43:14.508637 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:43:14.508637 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:43:14.508637 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:43:14.508637 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 11 00:43:14.508637 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 11 00:43:14.508637 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 11 00:43:14.508637 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 11 00:43:14.508637 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:43:14.508637 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:43:14.508637 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:43:14.508637 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:43:14.508637 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:43:14.508637 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:43:14.509558 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:43:14.509558 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:43:14.509558 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:43:14.509558 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:43:14.509558 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:43:14.509558 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:43:14.509558 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:43:14.509558 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:43:14.509558 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:43:14.509558 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:43:14.509558 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 11 00:43:14.509558 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 11 00:43:14.509558 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 11 00:43:14.509558 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:43:14.509558 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 656/1298] 10.222.0.27 () {56 vars in 1117 bytes} [Sun Sep 11 00:43:14 2022] GET /identity/v3/services/placement => generated 90 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:43:14.514977 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b5893a1-0971-4d51-ad0e-32cc74e134ea None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:43:14.535281 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:43:14.617728 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b5893a1-0971-4d51-ad0e-32cc74e134ea None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:43:14.651594 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8c099af-6ad7-46ac-a276-696492e4d1b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:43:14.652195 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c099af-6ad7-46ac-a276-696492e4d1b7 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:43:14.652474 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c099af-6ad7-46ac-a276-696492e4d1b7 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:43:14.652743 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c099af-6ad7-46ac-a276-696492e4d1b7 None admin] PATH_INFO: `/v3/services` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:43:14.653887 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8c099af-6ad7-46ac-a276-696492e4d1b7 None admin] RBAC: Adding query filter params (name=placement) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:43:14.654185 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8c099af-6ad7-46ac-a276-696492e4d1b7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:43:14.655574 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8c099af-6ad7-46ac-a276-696492e4d1b7 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:43:14.660787 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 643/1299] 10.222.0.27 () {56 vars in 1116 bytes} [Sun Sep 11 00:43:14 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:43:14.666697 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f51d3e12-b14b-4794-935c-bdff182fc9e4 None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:43:14.697512 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:43:14.764329 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f51d3e12-b14b-4794-935c-bdff182fc9e4 None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:43:14.797335 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40b63664-21a6-418b-a33a-7b4a5a50615b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:43:14.797932 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b63664-21a6-418b-a33a-7b4a5a50615b None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:43:14.798213 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b63664-21a6-418b-a33a-7b4a5a50615b None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:43:14.798486 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b63664-21a6-418b-a33a-7b4a5a50615b None admin] PATH_INFO: `/v3/endpoints` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:43:14.799209 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40b63664-21a6-418b-a33a-7b4a5a50615b None admin] RBAC: Adding query filter params (interface=public, service_id=8722f12373ad448b8fddc499ba2d8ec9, region_id=RegionOne) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:43:14.799519 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40b63664-21a6-418b-a33a-7b4a5a50615b None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:43:14.800878 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40b63664-21a6-418b-a33a-7b4a5a50615b None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:43:14.805737 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 657/1300] 10.222.0.27 () {56 vars in 1251 bytes} [Sun Sep 11 00:43:14 2022] GET /identity/v3/endpoints?service_id=8722f12373ad448b8fddc499ba2d8ec9&interface=public®ion_id=RegionOne => generated 194 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:43:14.811465 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8c099af-6ad7-46ac-a276-696492e4d1b7 None admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:43:14.890902 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:43:14.909720 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8c099af-6ad7-46ac-a276-696492e4d1b7 None admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:43:14.942484 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aa629e8-7a57-4c13-888b-a50490a57ddb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:43:14.943102 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa629e8-7a57-4c13-888b-a50490a57ddb None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:43:14.943380 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa629e8-7a57-4c13-888b-a50490a57ddb None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:43:14.943650 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa629e8-7a57-4c13-888b-a50490a57ddb None admin] PATH_INFO: `/v3/services` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:43:14.944248 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aa629e8-7a57-4c13-888b-a50490a57ddb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:43:14.945604 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aa629e8-7a57-4c13-888b-a50490a57ddb None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:43:14.952649 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 644/1301] 10.222.0.27 () {56 vars in 1087 bytes} [Sun Sep 11 00:43:14 2022] GET /identity/v3/services => generated 2191 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:43:17.098494 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2564b38-05d3-462a-b958-e168d98d5f4e None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:43:17.099048 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2564b38-05d3-462a-b958-e168d98d5f4e None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:43:17.099048 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2564b38-05d3-462a-b958-e168d98d5f4e None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:43:17.100403 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 658/1302] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:43:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:43:17.105769 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4132a1c3-9022-4dae-9a13-6df7f07da4e3 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:43:17.106268 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4132a1c3-9022-4dae-9a13-6df7f07da4e3 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:43:17.107045 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4132a1c3-9022-4dae-9a13-6df7f07da4e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:43:17.125307 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-4132a1c3-9022-4dae-9a13-6df7f07da4e3 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:43:17.126488 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-4132a1c3-9022-4dae-9a13-6df7f07da4e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:43:17.191149 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 645/1303] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:43:17 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 11 00:43:17.460227 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2564b38-05d3-462a-b958-e168d98d5f4e None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:43:17.565614 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:43:17.572895 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2564b38-05d3-462a-b958-e168d98d5f4e None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:43:17.606249 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e21f0f4-2585-4db0-af28-fc87a514454d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:43:17.606887 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e21f0f4-2585-4db0-af28-fc87a514454d None admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:43:17.607168 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e21f0f4-2585-4db0-af28-fc87a514454d None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:43:17.607436 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e21f0f4-2585-4db0-af28-fc87a514454d None admin] PATH_INFO: `/v3/services/placement` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:43:17.608019 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e21f0f4-2585-4db0-af28-fc87a514454d None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:43:17.609371 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e21f0f4-2585-4db0-af28-fc87a514454d None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:43:17.618231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application [None req-9e21f0f4-2585-4db0-af28-fc87a514454d None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 11 00:43:17.618231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:43:17.618231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 11 00:43:17.618231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 11 00:43:17.618231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 11 00:43:17.618231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 11 00:43:17.618231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 11 00:43:17.618231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 11 00:43:17.618231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 11 00:43:17.618231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:43:17.618231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 11 00:43:17.618231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:43:17.618231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 11 00:43:17.618231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 11 00:43:17.618231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 11 00:43:17.618231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 11 00:43:17.618231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 11 00:43:17.619457 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 11 00:43:17.619457 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 11 00:43:17.619457 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 11 00:43:17.619457 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 11 00:43:17.619457 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 11 00:43:17.619457 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 11 00:43:17.619457 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 11 00:43:17.619457 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 11 00:43:17.619457 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 11 00:43:17.619457 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 11 00:43:17.619457 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 11 00:43:17.619457 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 11 00:43:17.619457 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 11 00:43:17.619457 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 11 00:43:17.619457 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 11 00:43:17.619457 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.get_or_create( Sep 11 00:43:17.620323 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 11 00:43:17.620323 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application with Lock( Sep 11 00:43:17.620323 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 11 00:43:17.620323 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self._enter() Sep 11 00:43:17.620323 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 11 00:43:17.620323 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 11 00:43:17.620323 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 11 00:43:17.620323 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application return self.creator() Sep 11 00:43:17.620323 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 11 00:43:17.620323 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application created_value = creator( Sep 11 00:43:17.620323 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 11 00:43:17.620323 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 11 00:43:17.620323 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 11 00:43:17.620323 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: ERROR keystone.server.flask.application Sep 11 00:43:17.620323 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 659/1304] 10.222.0.27 () {56 vars in 1117 bytes} [Sun Sep 11 00:43:17 2022] GET /identity/v3/services/placement => generated 90 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 11 00:43:17.625757 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4132a1c3-9022-4dae-9a13-6df7f07da4e3 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:43:17.702251 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:43:17.725194 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4132a1c3-9022-4dae-9a13-6df7f07da4e3 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:43:17.758266 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a2579ce-0495-4eb8-9030-b45dc1d05415 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:43:17.758884 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2579ce-0495-4eb8-9030-b45dc1d05415 None admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:43:17.759165 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2579ce-0495-4eb8-9030-b45dc1d05415 None admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:43:17.759436 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2579ce-0495-4eb8-9030-b45dc1d05415 None admin] PATH_INFO: `/v3/services` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:43:17.760221 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a2579ce-0495-4eb8-9030-b45dc1d05415 None admin] RBAC: Adding query filter params (name=placement) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:43:17.760640 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a2579ce-0495-4eb8-9030-b45dc1d05415 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:43:17.763241 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a2579ce-0495-4eb8-9030-b45dc1d05415 None admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:43:17.769236 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 646/1305] 10.222.0.27 () {56 vars in 1116 bytes} [Sun Sep 11 00:43:17 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:43:17.775016 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e21f0f4-2585-4db0-af28-fc87a514454d None admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:43:17.871165 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e21f0f4-2585-4db0-af28-fc87a514454d None admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:43:17.907523 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ac5f7ea-8383-4d81-9465-be3aa1427da4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:43:17.908376 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac5f7ea-8383-4d81-9465-be3aa1427da4 None admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:43:17.908654 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac5f7ea-8383-4d81-9465-be3aa1427da4 None admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:43:17.908924 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac5f7ea-8383-4d81-9465-be3aa1427da4 None admin] PATH_INFO: `/v3/endpoints` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:43:17.909595 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ac5f7ea-8383-4d81-9465-be3aa1427da4 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:43:17.910979 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ac5f7ea-8383-4d81-9465-be3aa1427da4 None admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:43:17.931075 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 660/1306] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:43:17 2022] POST /identity/v3/endpoints => generated 331 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:43:27.765936 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=30 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:43:27.926662 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=91,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:44:22.300655 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6d641c-b46b-4ad4-b6ab-5f6db6e9b720 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:44:22.301129 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6d641c-b46b-4ad4-b6ab-5f6db6e9b720 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:44:22.301362 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6d641c-b46b-4ad4-b6ab-5f6db6e9b720 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:44:22.302829 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 647/1307] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:44:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:44:22.310434 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b108d216-c2e4-4683-8bec-5eb0dbdb3038 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:44:22.310871 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b108d216-c2e4-4683-8bec-5eb0dbdb3038 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:44:22.311165 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b108d216-c2e4-4683-8bec-5eb0dbdb3038 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:44:22.424049 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-b108d216-c2e4-4683-8bec-5eb0dbdb3038 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:44:22.425431 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-b108d216-c2e4-4683-8bec-5eb0dbdb3038 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:44:22.567664 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 661/1308] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:44:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 259 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:44:22.874352 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd43c8d4-6a75-4c1f-934c-55bf85c06c50 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:44:22.874712 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd43c8d4-6a75-4c1f-934c-55bf85c06c50 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:44:22.874945 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd43c8d4-6a75-4c1f-934c-55bf85c06c50 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:44:22.876431 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 648/1309] 10.222.0.27 () {54 vars in 965 bytes} [Sun Sep 11 00:44:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:44:22.883271 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1488a57b-a1c5-4d20-9d68-ac31ae25bb8c None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:44:22.883568 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1488a57b-a1c5-4d20-9d68-ac31ae25bb8c None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:44:22.884536 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1488a57b-a1c5-4d20-9d68-ac31ae25bb8c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:44:22.977057 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-1488a57b-a1c5-4d20-9d68-ac31ae25bb8c None None] MFA Rules not processed for user `a5f7b13283cb4ebfb25160aa6ad0e2f4`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:44:22.979177 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-1488a57b-a1c5-4d20-9d68-ac31ae25bb8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:44:23.002605 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:44:23.063717 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 662/1310] 10.222.0.27 () {58 vars in 1062 bytes} [Sun Sep 11 00:44:22 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:44:23.073417 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd43c8d4-6a75-4c1f-934c-55bf85c06c50 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:44:23.075337 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-dd43c8d4-6a75-4c1f-934c-55bf85c06c50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:44:23.196912 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd43c8d4-6a75-4c1f-934c-55bf85c06c50 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:44:23.287755 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e462b748-c541-408a-afce-14f009112ef7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:44:23.288360 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e462b748-c541-408a-afce-14f009112ef7 service neutron] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:44:23.288631 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e462b748-c541-408a-afce-14f009112ef7 service neutron] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:44:23.288895 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e462b748-c541-408a-afce-14f009112ef7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:44:23.291131 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-e462b748-c541-408a-afce-14f009112ef7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:44:23.349461 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:44:23.408476 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e462b748-c541-408a-afce-14f009112ef7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:44:23.410323 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e462b748-c541-408a-afce-14f009112ef7 service neutron] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:44:23.429391 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 649/1311] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:44:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 356 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:44:25.898487 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9690010-897e-48df-931f-770059b8c667 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:44:25.899064 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9690010-897e-48df-931f-770059b8c667 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:44:25.899064 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9690010-897e-48df-931f-770059b8c667 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:44:25.901173 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 663/1312] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:44:25 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:44:25.906957 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc2ef2b-f94e-4912-bbd0-493feb7451d7 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:44:25.907349 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc2ef2b-f94e-4912-bbd0-493feb7451d7 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:44:25.907522 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc2ef2b-f94e-4912-bbd0-493feb7451d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:44:25.977666 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-6cc2ef2b-f94e-4912-bbd0-493feb7451d7 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:44:25.978974 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-6cc2ef2b-f94e-4912-bbd0-493feb7451d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:44:26.073952 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 650/1313] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:44:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:44:26.357395 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9690010-897e-48df-931f-770059b8c667 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:44:26.358962 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-d9690010-897e-48df-931f-770059b8c667 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:44:26.464304 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9690010-897e-48df-931f-770059b8c667 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:44:26.537936 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:44:26.540955 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4394528a-47f5-4280-b6fd-03e616aaeb60 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:44:26.541755 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4394528a-47f5-4280-b6fd-03e616aaeb60 service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:44:26.542186 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4394528a-47f5-4280-b6fd-03e616aaeb60 service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:44:26.542649 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4394528a-47f5-4280-b6fd-03e616aaeb60 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:44:26.544279 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-4394528a-47f5-4280-b6fd-03e616aaeb60 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:44:26.655856 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4394528a-47f5-4280-b6fd-03e616aaeb60 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:44:26.657438 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4394528a-47f5-4280-b6fd-03e616aaeb60 service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:44:26.678563 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 664/1314] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:44:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:44:29.184742 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb4033b3-625f-421b-bcde-915a626eee41 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:44:29.185272 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb4033b3-625f-421b-bcde-915a626eee41 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:44:29.185272 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb4033b3-625f-421b-bcde-915a626eee41 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:44:29.186609 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 651/1315] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:44:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:44:29.191723 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93a038b-7f94-44ba-816e-b539c84b8e08 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:44:29.192155 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93a038b-7f94-44ba-816e-b539c84b8e08 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:44:29.192249 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93a038b-7f94-44ba-816e-b539c84b8e08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:44:29.258102 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-d93a038b-7f94-44ba-816e-b539c84b8e08 None None] MFA Rules not processed for user `2a97af87092c4282aa5985d5f454f26b`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:44:29.259525 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-d93a038b-7f94-44ba-816e-b539c84b8e08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:44:29.311717 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:44:29.356495 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 665/1316] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:44:29 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:44:29.643824 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb4033b3-625f-421b-bcde-915a626eee41 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:44:29.645364 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-eb4033b3-625f-421b-bcde-915a626eee41 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:44:29.655957 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:44:29.749841 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb4033b3-625f-421b-bcde-915a626eee41 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:44:29.822839 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410744dd-c62f-49fa-85a0-bb346151d138 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:44:29.823430 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410744dd-c62f-49fa-85a0-bb346151d138 service neutron] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:44:29.823706 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410744dd-c62f-49fa-85a0-bb346151d138 service neutron] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:44:29.823967 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410744dd-c62f-49fa-85a0-bb346151d138 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:44:29.825510 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-410744dd-c62f-49fa-85a0-bb346151d138 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:44:29.888167 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:44:29.936017 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-410744dd-c62f-49fa-85a0-bb346151d138 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:44:29.937456 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-410744dd-c62f-49fa-85a0-bb346151d138 service neutron] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:44:29.955504 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 652/1317] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:44:29 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:44:32.672967 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4240e599-864f-4249-b14d-68b3111e19df None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:44:32.673519 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4240e599-864f-4249-b14d-68b3111e19df None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:44:32.673782 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4240e599-864f-4249-b14d-68b3111e19df None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:44:32.675220 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 666/1318] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:44:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:44:32.681648 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01636db1-6607-4021-bf7e-6ebe18788506 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:44:32.682088 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01636db1-6607-4021-bf7e-6ebe18788506 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:44:32.682178 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01636db1-6607-4021-bf7e-6ebe18788506 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:44:32.757351 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-01636db1-6607-4021-bf7e-6ebe18788506 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:44:32.759154 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-01636db1-6607-4021-bf7e-6ebe18788506 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:44:32.865439 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 653/1319] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:44:32 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:44:33.178258 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4240e599-864f-4249-b14d-68b3111e19df None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:44:33.179811 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-4240e599-864f-4249-b14d-68b3111e19df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:44:33.282412 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4240e599-864f-4249-b14d-68b3111e19df None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:44:33.356859 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5940745a-7efa-4fb3-b014-68d7a0c17c3d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:44:33.357377 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5940745a-7efa-4fb3-b014-68d7a0c17c3d service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:44:33.357650 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5940745a-7efa-4fb3-b014-68d7a0c17c3d service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:44:33.357914 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5940745a-7efa-4fb3-b014-68d7a0c17c3d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:44:33.359505 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-5940745a-7efa-4fb3-b014-68d7a0c17c3d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:44:33.366273 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:44:33.468571 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5940745a-7efa-4fb3-b014-68d7a0c17c3d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:44:33.470002 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5940745a-7efa-4fb3-b014-68d7a0c17c3d service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:44:33.489544 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 667/1320] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:44:33 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:44:36.315693 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41068702-61e9-473e-b028-8687f719dc2a None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:44:36.316237 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41068702-61e9-473e-b028-8687f719dc2a None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:44:36.316237 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41068702-61e9-473e-b028-8687f719dc2a None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:44:36.317634 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 654/1321] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:44:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:44:36.324215 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c66263e5-ad56-4bdd-aa83-f4430374d989 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:44:36.324770 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c66263e5-ad56-4bdd-aa83-f4430374d989 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:44:36.324770 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c66263e5-ad56-4bdd-aa83-f4430374d989 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:44:36.398089 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-c66263e5-ad56-4bdd-aa83-f4430374d989 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:44:36.399824 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-c66263e5-ad56-4bdd-aa83-f4430374d989 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:44:36.462249 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:44:36.504746 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 668/1322] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:44:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:44:36.800638 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f334046e-de66-40a2-b6cf-0cae1f687547 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:44:36.800924 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f334046e-de66-40a2-b6cf-0cae1f687547 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:44:36.801203 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f334046e-de66-40a2-b6cf-0cae1f687547 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:44:36.802586 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 655/1323] 10.222.0.27 () {54 vars in 965 bytes} [Sun Sep 11 00:44:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:44:36.809002 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-433ef524-69f0-4fb4-92fb-1e822cccf5f8 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:44:36.809364 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-433ef524-69f0-4fb4-92fb-1e822cccf5f8 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:44:36.809646 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-433ef524-69f0-4fb4-92fb-1e822cccf5f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:44:36.886023 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-433ef524-69f0-4fb4-92fb-1e822cccf5f8 None None] MFA Rules not processed for user `a5f7b13283cb4ebfb25160aa6ad0e2f4`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:44:36.887267 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-433ef524-69f0-4fb4-92fb-1e822cccf5f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:44:36.965589 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 669/1324] 10.222.0.27 () {58 vars in 1062 bytes} [Sun Sep 11 00:44:36 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:44:36.974374 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f334046e-de66-40a2-b6cf-0cae1f687547 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:44:36.976003 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-f334046e-de66-40a2-b6cf-0cae1f687547 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:44:36.996573 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:44:37.082149 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f334046e-de66-40a2-b6cf-0cae1f687547 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:44:37.158077 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-291ecb4e-4c24-4b7f-b65a-e09c758e0205 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:44:37.158669 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-291ecb4e-4c24-4b7f-b65a-e09c758e0205 service neutron] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:44:37.158970 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-291ecb4e-4c24-4b7f-b65a-e09c758e0205 service neutron] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:44:37.159238 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-291ecb4e-4c24-4b7f-b65a-e09c758e0205 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:44:37.160793 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-291ecb4e-4c24-4b7f-b65a-e09c758e0205 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:44:37.238400 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:44:37.276991 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-291ecb4e-4c24-4b7f-b65a-e09c758e0205 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:44:37.278428 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-291ecb4e-4c24-4b7f-b65a-e09c758e0205 service neutron] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:44:37.297254 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 656/1325] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:44:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:44:39.755373 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f4e849-f659-4fa2-bd0c-1e3e36ac4625 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:44:39.755944 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f4e849-f659-4fa2-bd0c-1e3e36ac4625 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:44:39.755944 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f4e849-f659-4fa2-bd0c-1e3e36ac4625 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:44:39.757246 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 670/1326] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:44:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:44:39.763708 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc60c39f-015e-4a96-b871-a6d747d4c671 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:44:39.764143 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc60c39f-015e-4a96-b871-a6d747d4c671 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:44:39.764260 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc60c39f-015e-4a96-b871-a6d747d4c671 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:44:39.835529 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-fc60c39f-015e-4a96-b871-a6d747d4c671 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:44:39.837015 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-fc60c39f-015e-4a96-b871-a6d747d4c671 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:44:39.935267 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 657/1327] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:44:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:44:40.216645 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f4e849-f659-4fa2-bd0c-1e3e36ac4625 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:44:40.218171 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-a1f4e849-f659-4fa2-bd0c-1e3e36ac4625 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:44:40.281221 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:44:40.326071 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f4e849-f659-4fa2-bd0c-1e3e36ac4625 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:44:40.400803 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c8e9a55-ea17-41ae-933f-74a854bab333 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:44:40.401322 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c8e9a55-ea17-41ae-933f-74a854bab333 service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:44:40.401593 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c8e9a55-ea17-41ae-933f-74a854bab333 service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:44:40.401857 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c8e9a55-ea17-41ae-933f-74a854bab333 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:44:40.403452 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-6c8e9a55-ea17-41ae-933f-74a854bab333 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:44:40.511915 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:44:40.514372 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c8e9a55-ea17-41ae-933f-74a854bab333 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:44:40.516281 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c8e9a55-ea17-41ae-933f-74a854bab333 service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:44:40.536381 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 671/1328] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:44:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:44:42.757025 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d253407-c13f-49a2-8bcd-d298922362b4 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:44:42.757524 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d253407-c13f-49a2-8bcd-d298922362b4 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:44:42.757642 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d253407-c13f-49a2-8bcd-d298922362b4 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:44:42.759607 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 658/1329] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:44:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:44:42.765105 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fe399f-a02f-46f1-8746-304ffcf1f75d None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:44:42.765578 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fe399f-a02f-46f1-8746-304ffcf1f75d None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:44:42.765671 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fe399f-a02f-46f1-8746-304ffcf1f75d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:44:42.836628 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-d9fe399f-a02f-46f1-8746-304ffcf1f75d None None] MFA Rules not processed for user `2a97af87092c4282aa5985d5f454f26b`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:44:42.838150 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-d9fe399f-a02f-46f1-8746-304ffcf1f75d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:44:42.937532 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 672/1330] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:44:42 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:44:43.219376 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d253407-c13f-49a2-8bcd-d298922362b4 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:44:43.220899 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-0d253407-c13f-49a2-8bcd-d298922362b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:44:43.251613 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:44:43.327996 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d253407-c13f-49a2-8bcd-d298922362b4 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:44:43.404547 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc73873-b669-4476-a895-7d2fd37d8ef0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:44:43.405155 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc73873-b669-4476-a895-7d2fd37d8ef0 service neutron] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:44:43.405431 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc73873-b669-4476-a895-7d2fd37d8ef0 service neutron] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:44:43.405698 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc73873-b669-4476-a895-7d2fd37d8ef0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:44:43.407602 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-2dc73873-b669-4476-a895-7d2fd37d8ef0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:44:43.489258 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:44:43.529378 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc73873-b669-4476-a895-7d2fd37d8ef0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:44:43.530870 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc73873-b669-4476-a895-7d2fd37d8ef0 service neutron] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:44:43.556833 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 659/1331] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:44:43 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:44:45.939846 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0a0d67-9aaf-4701-90ba-04f4b846b193 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:44:45.940375 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0a0d67-9aaf-4701-90ba-04f4b846b193 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:44:45.940375 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0a0d67-9aaf-4701-90ba-04f4b846b193 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:44:45.941655 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 673/1332] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:44:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:44:45.947571 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69faea7c-7d06-49cc-8bef-8803a29bb18b None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:44:45.947985 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69faea7c-7d06-49cc-8bef-8803a29bb18b None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:44:45.948179 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69faea7c-7d06-49cc-8bef-8803a29bb18b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:44:46.013089 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-69faea7c-7d06-49cc-8bef-8803a29bb18b None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:44:46.014348 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-69faea7c-7d06-49cc-8bef-8803a29bb18b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:44:46.112187 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 660/1333] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:44:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:44:46.395227 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b0a0d67-9aaf-4701-90ba-04f4b846b193 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:44:46.396791 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-6b0a0d67-9aaf-4701-90ba-04f4b846b193 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:44:46.466811 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:44:46.500186 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b0a0d67-9aaf-4701-90ba-04f4b846b193 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:44:46.574466 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-248fea4d-3dea-4f19-8258-05bdab58d142 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:44:46.575096 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248fea4d-3dea-4f19-8258-05bdab58d142 service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:44:46.575377 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248fea4d-3dea-4f19-8258-05bdab58d142 service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:44:46.575647 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248fea4d-3dea-4f19-8258-05bdab58d142 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:44:46.577221 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-248fea4d-3dea-4f19-8258-05bdab58d142 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:44:46.685246 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-248fea4d-3dea-4f19-8258-05bdab58d142 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:44:46.686597 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-248fea4d-3dea-4f19-8258-05bdab58d142 service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:44:46.697182 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:44:46.705820 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 674/1334] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:44:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:44:48.914291 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7494fa-ccbb-405f-9825-ab7f68ddd6b2 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:44:48.914870 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7494fa-ccbb-405f-9825-ab7f68ddd6b2 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:44:48.915018 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7494fa-ccbb-405f-9825-ab7f68ddd6b2 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:44:48.916363 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 661/1335] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:44:48 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:44:48.921540 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b58a5f6-eb40-4ca4-a09f-6983fa344cfb None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:44:48.921984 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b58a5f6-eb40-4ca4-a09f-6983fa344cfb None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:44:48.922082 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b58a5f6-eb40-4ca4-a09f-6983fa344cfb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:44:48.988473 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-7b58a5f6-eb40-4ca4-a09f-6983fa344cfb None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:44:48.989673 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-7b58a5f6-eb40-4ca4-a09f-6983fa344cfb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:44:49.083977 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 675/1336] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:44:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:44:49.374862 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c7494fa-ccbb-405f-9825-ab7f68ddd6b2 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:44:49.376383 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-1c7494fa-ccbb-405f-9825-ab7f68ddd6b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:44:49.417098 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:44:49.479751 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c7494fa-ccbb-405f-9825-ab7f68ddd6b2 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:44:49.554657 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a26e034-38dc-4b44-b50a-6de60f6d7117 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:44:49.555297 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a26e034-38dc-4b44-b50a-6de60f6d7117 service neutron] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:44:49.555568 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a26e034-38dc-4b44-b50a-6de60f6d7117 service neutron] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:44:49.555834 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a26e034-38dc-4b44-b50a-6de60f6d7117 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:44:49.557413 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-2a26e034-38dc-4b44-b50a-6de60f6d7117 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:44:49.640116 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:44:49.666674 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a26e034-38dc-4b44-b50a-6de60f6d7117 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:44:49.668140 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a26e034-38dc-4b44-b50a-6de60f6d7117 service neutron] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:44:49.687147 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 662/1337] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:44:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:44:52.491025 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5dad4f-eefb-47e2-855b-7adc3dd19a8c None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:44:52.492130 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5dad4f-eefb-47e2-855b-7adc3dd19a8c None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:44:52.492130 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5dad4f-eefb-47e2-855b-7adc3dd19a8c None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:44:52.493535 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 676/1338] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:44:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:44:52.498602 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7baf3ea3-366f-4674-b12d-0c0234db482b None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:44:52.499046 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7baf3ea3-366f-4674-b12d-0c0234db482b None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:44:52.499169 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7baf3ea3-366f-4674-b12d-0c0234db482b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:44:52.567159 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-7baf3ea3-366f-4674-b12d-0c0234db482b None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:44:52.568364 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-7baf3ea3-366f-4674-b12d-0c0234db482b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:44:52.664450 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 663/1339] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:44:52 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:44:52.947522 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5dad4f-eefb-47e2-855b-7adc3dd19a8c None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:44:52.949044 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-1e5dad4f-eefb-47e2-855b-7adc3dd19a8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:44:53.023963 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:44:53.051210 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5dad4f-eefb-47e2-855b-7adc3dd19a8c None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:44:53.123922 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c6f8c4-244a-43a1-a2fd-fec6d53035b2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:44:53.124532 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c6f8c4-244a-43a1-a2fd-fec6d53035b2 service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:44:53.124811 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c6f8c4-244a-43a1-a2fd-fec6d53035b2 service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:44:53.125080 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c6f8c4-244a-43a1-a2fd-fec6d53035b2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:44:53.126659 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-44c6f8c4-244a-43a1-a2fd-fec6d53035b2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:44:53.235642 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44c6f8c4-244a-43a1-a2fd-fec6d53035b2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:44:53.237069 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44c6f8c4-244a-43a1-a2fd-fec6d53035b2 service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:44:53.255883 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 677/1340] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:44:52 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:44:57.484651 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7aeffe-cc64-4e9f-83f1-1be0dc249e71 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:44:57.485285 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7aeffe-cc64-4e9f-83f1-1be0dc249e71 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:44:57.485405 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7aeffe-cc64-4e9f-83f1-1be0dc249e71 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:44:57.487409 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 664/1341] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:44:57 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:44:57.494227 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cca5be3-cfaa-48a4-bcb7-33f609b5c455 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:44:57.494648 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cca5be3-cfaa-48a4-bcb7-33f609b5c455 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:44:57.495013 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cca5be3-cfaa-48a4-bcb7-33f609b5c455 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:44:57.500487 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:44:57.561164 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-3cca5be3-cfaa-48a4-bcb7-33f609b5c455 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:44:57.562288 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-3cca5be3-cfaa-48a4-bcb7-33f609b5c455 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:44:57.658799 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 678/1342] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:44:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:44:57.953412 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7aeffe-cc64-4e9f-83f1-1be0dc249e71 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:44:57.954954 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-9a7aeffe-cc64-4e9f-83f1-1be0dc249e71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:44:58.004482 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:44:58.062816 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7aeffe-cc64-4e9f-83f1-1be0dc249e71 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:44:58.142237 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e39820a6-62f0-4ec0-bb1e-5e8ec5a38fbf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:44:58.142874 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39820a6-62f0-4ec0-bb1e-5e8ec5a38fbf service neutron] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:44:58.143160 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39820a6-62f0-4ec0-bb1e-5e8ec5a38fbf service neutron] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:44:58.143427 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39820a6-62f0-4ec0-bb1e-5e8ec5a38fbf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:44:58.145016 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-e39820a6-62f0-4ec0-bb1e-5e8ec5a38fbf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:44:58.238614 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:44:58.255651 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e39820a6-62f0-4ec0-bb1e-5e8ec5a38fbf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:44:58.257644 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e39820a6-62f0-4ec0-bb1e-5e8ec5a38fbf service neutron] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:44:58.277087 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 665/1343] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:44:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:45:01.057913 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a321186-4fd0-4e78-b9ff-6a2096d3c6c1 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:01.058460 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a321186-4fd0-4e78-b9ff-6a2096d3c6c1 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:01.058460 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a321186-4fd0-4e78-b9ff-6a2096d3c6c1 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:01.060009 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 679/1344] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:45:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:45:01.065153 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748b64c9-9363-41af-85a6-c784ae48f4c4 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:01.065622 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748b64c9-9363-41af-85a6-c784ae48f4c4 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:01.065733 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748b64c9-9363-41af-85a6-c784ae48f4c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:01.132054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-748b64c9-9363-41af-85a6-c784ae48f4c4 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:45:01.133255 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-748b64c9-9363-41af-85a6-c784ae48f4c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:01.229536 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 666/1345] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:45:01 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:45:01.512488 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a321186-4fd0-4e78-b9ff-6a2096d3c6c1 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:45:01.514001 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-7a321186-4fd0-4e78-b9ff-6a2096d3c6c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:01.605527 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:45:01.625198 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a321186-4fd0-4e78-b9ff-6a2096d3c6c1 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:45:01.702004 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b042847-6d61-47d0-93cd-02e661928029 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:45:01.702611 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b042847-6d61-47d0-93cd-02e661928029 service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:01.702928 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b042847-6d61-47d0-93cd-02e661928029 service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:01.703202 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b042847-6d61-47d0-93cd-02e661928029 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:01.704788 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-9b042847-6d61-47d0-93cd-02e661928029 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:01.817964 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b042847-6d61-47d0-93cd-02e661928029 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:45:01.820107 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b042847-6d61-47d0-93cd-02e661928029 service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:45:01.840069 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 680/1346] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:45:01 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 328 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:45:06.245896 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f184220a-41d7-408a-baf5-250ba8a387cc None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:06.246363 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f184220a-41d7-408a-baf5-250ba8a387cc None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:06.246469 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f184220a-41d7-408a-baf5-250ba8a387cc None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:06.247834 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 667/1347] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:45:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:45:06.253412 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b2fff9-7ae3-4975-af4d-ef42a2e33877 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:06.253932 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b2fff9-7ae3-4975-af4d-ef42a2e33877 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:06.253932 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b2fff9-7ae3-4975-af4d-ef42a2e33877 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:06.270341 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:45:06.324336 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-e7b2fff9-7ae3-4975-af4d-ef42a2e33877 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:45:06.325545 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-e7b2fff9-7ae3-4975-af4d-ef42a2e33877 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:06.423807 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 681/1348] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:45:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:45:06.709338 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f184220a-41d7-408a-baf5-250ba8a387cc None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:45:06.711456 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-f184220a-41d7-408a-baf5-250ba8a387cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:06.767891 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:45:06.814658 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f184220a-41d7-408a-baf5-250ba8a387cc None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:45:06.887518 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8573108c-1622-4ddd-a8f5-0b6c6b4cb3b6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:45:06.888113 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8573108c-1622-4ddd-a8f5-0b6c6b4cb3b6 service neutron] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:06.888388 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8573108c-1622-4ddd-a8f5-0b6c6b4cb3b6 service neutron] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:06.888651 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8573108c-1622-4ddd-a8f5-0b6c6b4cb3b6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:06.890203 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-8573108c-1622-4ddd-a8f5-0b6c6b4cb3b6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:06.995643 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:45:07.000002 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8573108c-1622-4ddd-a8f5-0b6c6b4cb3b6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:45:07.001471 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8573108c-1622-4ddd-a8f5-0b6c6b4cb3b6 service neutron] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:45:07.020432 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 668/1349] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:45:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:45:11.996501 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b22c39-238b-4f8b-9563-0b561bb8e2b4 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:11.997016 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b22c39-238b-4f8b-9563-0b561bb8e2b4 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:11.997119 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b22c39-238b-4f8b-9563-0b561bb8e2b4 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:11.999617 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 682/1350] 10.222.0.27 () {54 vars in 921 bytes} [Sun Sep 11 00:45:11 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:45:12.005671 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b4bbc7-e34f-4482-98af-c31db73495f7 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:12.006664 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b4bbc7-e34f-4482-98af-c31db73495f7 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:12.006664 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b4bbc7-e34f-4482-98af-c31db73495f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:12.109727 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-67b4bbc7-e34f-4482-98af-c31db73495f7 None None] MFA Rules not processed for user `cdbfc2ccd73e4a5eb8c999be5b094619`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:45:12.117949 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-67b4bbc7-e34f-4482-98af-c31db73495f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:12.279786 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 669/1351] 10.222.0.27 () {58 vars in 1018 bytes} [Sun Sep 11 00:45:12 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 275 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:45:16.422248 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=59 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:45:16.526612 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b348156e-d8e5-439b-9bfe-5a293f1cdb60 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:16.526911 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b348156e-d8e5-439b-9bfe-5a293f1cdb60 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:16.527187 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b348156e-d8e5-439b-9bfe-5a293f1cdb60 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:16.528739 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 683/1352] 10.222.0.27 () {54 vars in 921 bytes} [Sun Sep 11 00:45:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:45:16.535623 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e9f654-def1-4480-b452-17da0da447ba None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:16.536084 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e9f654-def1-4480-b452-17da0da447ba None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:16.536365 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e9f654-def1-4480-b452-17da0da447ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:16.594559 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:45:16.619746 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-02e9f654-def1-4480-b452-17da0da447ba None None] MFA Rules not processed for user `cdbfc2ccd73e4a5eb8c999be5b094619`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:45:16.621115 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-02e9f654-def1-4480-b452-17da0da447ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:16.734303 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 670/1353] 10.222.0.27 () {58 vars in 1018 bytes} [Sun Sep 11 00:45:16 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:45:18.066773 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be98ee4-4e10-4203-a89b-42eec6c9301d None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:18.069877 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be98ee4-4e10-4203-a89b-42eec6c9301d None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:18.069877 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be98ee4-4e10-4203-a89b-42eec6c9301d None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:18.070450 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 684/1354] 10.222.0.27 () {54 vars in 921 bytes} [Sun Sep 11 00:45:18 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:45:18.076030 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd969cb-be2d-4343-a43a-0df6922d3251 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:18.076527 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd969cb-be2d-4343-a43a-0df6922d3251 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:18.076655 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd969cb-be2d-4343-a43a-0df6922d3251 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:18.159067 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-cbd969cb-be2d-4343-a43a-0df6922d3251 None None] MFA Rules not processed for user `cdbfc2ccd73e4a5eb8c999be5b094619`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:45:18.160282 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-cbd969cb-be2d-4343-a43a-0df6922d3251 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:18.208496 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:45:18.275161 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 671/1355] 10.222.0.27 () {58 vars in 1018 bytes} [Sun Sep 11 00:45:18 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:45:21.860166 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b4663e-888b-47e4-8422-4cd1dd1e6d46 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:21.861143 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b4663e-888b-47e4-8422-4cd1dd1e6d46 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:21.861630 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b4663e-888b-47e4-8422-4cd1dd1e6d46 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:21.863194 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 685/1356] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:45:21 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:45:21.868916 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff7f0eb-bdbd-42c5-9e0e-a6285e805e53 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:21.869444 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff7f0eb-bdbd-42c5-9e0e-a6285e805e53 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:21.869444 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff7f0eb-bdbd-42c5-9e0e-a6285e805e53 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:21.941191 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-eff7f0eb-bdbd-42c5-9e0e-a6285e805e53 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:45:21.942365 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-eff7f0eb-bdbd-42c5-9e0e-a6285e805e53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:22.042601 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 672/1357] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:45:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:45:22.363056 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037e1983-9150-4889-abf5-53895e92a5e8 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:22.363526 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037e1983-9150-4889-abf5-53895e92a5e8 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:22.363974 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037e1983-9150-4889-abf5-53895e92a5e8 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:22.366345 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 686/1358] 10.222.0.27 () {54 vars in 962 bytes} [Sun Sep 11 00:45:22 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:45:22.375230 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7483c960-ff58-4f92-8982-8eb75af2de2f None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:22.375833 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7483c960-ff58-4f92-8982-8eb75af2de2f None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:22.376357 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7483c960-ff58-4f92-8982-8eb75af2de2f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:22.400213 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:45:22.464838 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-7483c960-ff58-4f92-8982-8eb75af2de2f None None] MFA Rules not processed for user `835def0e6c6d4365a2767051509eb0d6`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:45:22.466160 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-7483c960-ff58-4f92-8982-8eb75af2de2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:22.609326 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 673/1359] 10.222.0.27 () {58 vars in 1059 bytes} [Sun Sep 11 00:45:22 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 236 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:45:22.618218 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-037e1983-9150-4889-abf5-53895e92a5e8 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:45:22.619760 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-037e1983-9150-4889-abf5-53895e92a5e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:22.782585 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-037e1983-9150-4889-abf5-53895e92a5e8 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:45:22.872700 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:45:22.917695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d04170d-30d8-42dd-8ada-4f5788fbefd9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '835def0e6c6d4365a2767051509eb0d6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'ResellerAdmin', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:45:22.918500 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d04170d-30d8-42dd-8ada-4f5788fbefd9 service nova] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:22.918905 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d04170d-30d8-42dd-8ada-4f5788fbefd9 service nova] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:22.919272 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d04170d-30d8-42dd-8ada-4f5788fbefd9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:22.921110 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-5d04170d-30d8-42dd-8ada-4f5788fbefd9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:23.025386 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d04170d-30d8-42dd-8ada-4f5788fbefd9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:45:23.027178 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d04170d-30d8-42dd-8ada-4f5788fbefd9 service nova] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:45:23.047244 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 687/1360] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:45:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 429 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:45:25.459662 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cfbdd28-77be-4e3f-8838-f87347f6ff3b None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:25.459662 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cfbdd28-77be-4e3f-8838-f87347f6ff3b None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:25.460473 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cfbdd28-77be-4e3f-8838-f87347f6ff3b None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:25.463129 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 674/1361] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:45:25 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:45:25.466528 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef0adbf-a278-456d-a8d5-074477d254e4 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:25.467016 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef0adbf-a278-456d-a8d5-074477d254e4 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:25.467193 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef0adbf-a278-456d-a8d5-074477d254e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:25.539615 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-4ef0adbf-a278-456d-a8d5-074477d254e4 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:45:25.540926 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-4ef0adbf-a278-456d-a8d5-074477d254e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:25.580823 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:45:25.644690 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 688/1362] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:45:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:45:25.942572 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cfbdd28-77be-4e3f-8838-f87347f6ff3b None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:45:25.945154 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-7cfbdd28-77be-4e3f-8838-f87347f6ff3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:26.031156 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:45:26.139099 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cfbdd28-77be-4e3f-8838-f87347f6ff3b None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:45:26.262967 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a90759b3-0359-45b6-92fb-cef8db088a28 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '835def0e6c6d4365a2767051509eb0d6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'ResellerAdmin', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:45:26.263583 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90759b3-0359-45b6-92fb-cef8db088a28 service nova] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:26.263859 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90759b3-0359-45b6-92fb-cef8db088a28 service nova] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:26.264128 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90759b3-0359-45b6-92fb-cef8db088a28 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:26.265725 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-a90759b3-0359-45b6-92fb-cef8db088a28 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:26.291527 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:45:26.363039 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a90759b3-0359-45b6-92fb-cef8db088a28 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:45:26.365317 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a90759b3-0359-45b6-92fb-cef8db088a28 service nova] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:45:26.384418 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 675/1363] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:45:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 442 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:45:28.922028 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea0e3564-31f6-4b3b-9fd0-eb966f5dfaa2 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:28.922546 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea0e3564-31f6-4b3b-9fd0-eb966f5dfaa2 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:28.922546 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea0e3564-31f6-4b3b-9fd0-eb966f5dfaa2 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:28.923901 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 689/1364] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:45:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:45:28.929450 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4453ef5-77b0-4aea-bb0a-9da22e580e39 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:28.929896 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4453ef5-77b0-4aea-bb0a-9da22e580e39 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:28.929994 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4453ef5-77b0-4aea-bb0a-9da22e580e39 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:29.011408 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-e4453ef5-77b0-4aea-bb0a-9da22e580e39 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:45:29.012999 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-e4453ef5-77b0-4aea-bb0a-9da22e580e39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:29.141580 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:45:29.151206 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 676/1365] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:45:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 223 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:45:29.444614 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa1dda80-c0fa-422f-a815-457d273bf991 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:29.444898 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa1dda80-c0fa-422f-a815-457d273bf991 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:29.445171 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa1dda80-c0fa-422f-a815-457d273bf991 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:29.446631 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 690/1366] 10.222.0.27 () {54 vars in 962 bytes} [Sun Sep 11 00:45:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:45:29.451864 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af9f7f4-a3fc-4c54-9a2f-2d16e3a898b7 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:29.452220 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af9f7f4-a3fc-4c54-9a2f-2d16e3a898b7 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:29.452750 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af9f7f4-a3fc-4c54-9a2f-2d16e3a898b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:29.541428 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-0af9f7f4-a3fc-4c54-9a2f-2d16e3a898b7 None None] MFA Rules not processed for user `835def0e6c6d4365a2767051509eb0d6`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:45:29.542626 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-0af9f7f4-a3fc-4c54-9a2f-2d16e3a898b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:29.670542 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 677/1367] 10.222.0.27 () {58 vars in 1059 bytes} [Sun Sep 11 00:45:29 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 219 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:45:29.679784 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa1dda80-c0fa-422f-a815-457d273bf991 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:45:29.683033 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-aa1dda80-c0fa-422f-a815-457d273bf991 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:29.847623 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa1dda80-c0fa-422f-a815-457d273bf991 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:45:29.870707 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:45:29.970379 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab06c950-cbe2-4283-bc52-9ff8189570dc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '835def0e6c6d4365a2767051509eb0d6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'ResellerAdmin', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:45:29.971499 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab06c950-cbe2-4283-bc52-9ff8189570dc service nova] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:29.971499 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab06c950-cbe2-4283-bc52-9ff8189570dc service nova] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:29.971790 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab06c950-cbe2-4283-bc52-9ff8189570dc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:29.973204 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-ab06c950-cbe2-4283-bc52-9ff8189570dc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:30.075884 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab06c950-cbe2-4283-bc52-9ff8189570dc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:45:30.080128 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab06c950-cbe2-4283-bc52-9ff8189570dc service nova] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:45:30.097116 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 691/1368] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:45:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 418 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:45:30.133634 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7992ad27-6897-4afd-b41d-8a05380f8480 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:30.134175 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7992ad27-6897-4afd-b41d-8a05380f8480 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:30.134175 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7992ad27-6897-4afd-b41d-8a05380f8480 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:30.135851 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 678/1369] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:45:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:45:30.141970 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3267641b-d418-4e61-8368-e666677b02ea None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:30.142265 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3267641b-d418-4e61-8368-e666677b02ea None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:30.142532 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3267641b-d418-4e61-8368-e666677b02ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:30.176435 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:45:30.220260 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-3267641b-d418-4e61-8368-e666677b02ea None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:45:30.221722 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-3267641b-d418-4e61-8368-e666677b02ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:30.347649 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 692/1370] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:45:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:45:30.685087 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd7a9254-f352-42f3-9cf7-dd73dadc6e34 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:30.685087 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd7a9254-f352-42f3-9cf7-dd73dadc6e34 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:30.685087 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd7a9254-f352-42f3-9cf7-dd73dadc6e34 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:30.687811 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 679/1371] 10.222.0.27 () {54 vars in 964 bytes} [Sun Sep 11 00:45:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:45:30.691493 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064c9e3a-9ae9-48fe-bddd-d8af39bbf94e None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:30.691781 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064c9e3a-9ae9-48fe-bddd-d8af39bbf94e None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:30.692076 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064c9e3a-9ae9-48fe-bddd-d8af39bbf94e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:30.771789 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-064c9e3a-9ae9-48fe-bddd-d8af39bbf94e None None] MFA Rules not processed for user `22374eaff0834245b88506dcd7b24ede`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:45:30.772996 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-064c9e3a-9ae9-48fe-bddd-d8af39bbf94e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:30.824340 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:45:30.857172 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 693/1372] 10.222.0.27 () {58 vars in 1061 bytes} [Sun Sep 11 00:45:30 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:45:30.864689 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd7a9254-f352-42f3-9cf7-dd73dadc6e34 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:45:30.866449 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-cd7a9254-f352-42f3-9cf7-dd73dadc6e34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:30.921135 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:45:30.978674 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd7a9254-f352-42f3-9cf7-dd73dadc6e34 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:45:31.061848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5fc8a57-7e00-4f48-ada7-b2eae34ead72 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:45:31.062462 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fc8a57-7e00-4f48-ada7-b2eae34ead72 service cinder] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:31.062733 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fc8a57-7e00-4f48-ada7-b2eae34ead72 service cinder] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:31.063030 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fc8a57-7e00-4f48-ada7-b2eae34ead72 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:31.064615 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-c5fc8a57-7e00-4f48-ada7-b2eae34ead72 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:31.186969 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:45:31.203667 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5fc8a57-7e00-4f48-ada7-b2eae34ead72 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:45:31.205116 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5fc8a57-7e00-4f48-ada7-b2eae34ead72 service cinder] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:45:31.225920 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 680/1373] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:45:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 362 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:45:32.378255 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-104b438c-c68b-4df5-8cfc-eea6ba349572 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:32.378846 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-104b438c-c68b-4df5-8cfc-eea6ba349572 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:32.378846 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-104b438c-c68b-4df5-8cfc-eea6ba349572 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:32.380157 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 694/1374] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:45:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:45:32.386222 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d920c27b-3e99-4de8-a880-a6b2d49fb2d6 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:32.386636 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d920c27b-3e99-4de8-a880-a6b2d49fb2d6 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:32.386955 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d920c27b-3e99-4de8-a880-a6b2d49fb2d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:32.453470 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-d920c27b-3e99-4de8-a880-a6b2d49fb2d6 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:45:32.454649 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-d920c27b-3e99-4de8-a880-a6b2d49fb2d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:32.550051 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 681/1375] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:45:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:45:32.860204 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-104b438c-c68b-4df5-8cfc-eea6ba349572 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:45:32.861721 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-104b438c-c68b-4df5-8cfc-eea6ba349572 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:33.017664 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-104b438c-c68b-4df5-8cfc-eea6ba349572 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:45:33.072218 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:45:33.140842 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0017f136-65eb-4e9a-a559-b7a1c27c6c06 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '835def0e6c6d4365a2767051509eb0d6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'ResellerAdmin', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:45:33.141480 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0017f136-65eb-4e9a-a559-b7a1c27c6c06 service nova] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:33.141759 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0017f136-65eb-4e9a-a559-b7a1c27c6c06 service nova] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:33.142027 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0017f136-65eb-4e9a-a559-b7a1c27c6c06 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:33.143634 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-0017f136-65eb-4e9a-a559-b7a1c27c6c06 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:33.237228 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0017f136-65eb-4e9a-a559-b7a1c27c6c06 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:45:33.239391 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0017f136-65eb-4e9a-a559-b7a1c27c6c06 service nova] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:45:33.260104 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 695/1376] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:45:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 400 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:45:35.584345 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20bcbaf-05a2-4888-b48f-2462c4a81d96 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:35.584902 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20bcbaf-05a2-4888-b48f-2462c4a81d96 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:35.584902 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20bcbaf-05a2-4888-b48f-2462c4a81d96 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:35.586240 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 682/1377] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:45:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:45:35.591619 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130bd18c-a0be-4efa-ac33-1e8a780d855e None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:35.592070 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130bd18c-a0be-4efa-ac33-1e8a780d855e None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:35.592160 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130bd18c-a0be-4efa-ac33-1e8a780d855e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:35.662572 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-130bd18c-a0be-4efa-ac33-1e8a780d855e None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:45:35.663916 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-130bd18c-a0be-4efa-ac33-1e8a780d855e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:35.671316 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:45:35.774515 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 696/1378] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:45:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:45:36.068301 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e20bcbaf-05a2-4888-b48f-2462c4a81d96 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:45:36.070040 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-e20bcbaf-05a2-4888-b48f-2462c4a81d96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:36.139807 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:45:36.231690 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e20bcbaf-05a2-4888-b48f-2462c4a81d96 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:45:36.351433 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e565b0-6e39-44d0-9156-355f95e3168f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '835def0e6c6d4365a2767051509eb0d6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'ResellerAdmin', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:45:36.351947 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e565b0-6e39-44d0-9156-355f95e3168f service nova] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:36.352221 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e565b0-6e39-44d0-9156-355f95e3168f service nova] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:36.352488 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e565b0-6e39-44d0-9156-355f95e3168f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:36.354045 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-00e565b0-6e39-44d0-9156-355f95e3168f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:36.373627 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:45:36.451912 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00e565b0-6e39-44d0-9156-355f95e3168f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:45:36.453524 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00e565b0-6e39-44d0-9156-355f95e3168f service nova] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:45:36.472432 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 683/1379] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:45:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 404 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:45:38.724651 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1187ba7f-7ad9-4eb1-867a-41bdbb9c9e93 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:38.725239 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1187ba7f-7ad9-4eb1-867a-41bdbb9c9e93 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:38.725239 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1187ba7f-7ad9-4eb1-867a-41bdbb9c9e93 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:38.726656 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 697/1380] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:45:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:45:38.731826 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46794db-2c88-4421-89b1-3fd6ce8b2ade None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:38.732250 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46794db-2c88-4421-89b1-3fd6ce8b2ade None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:38.732430 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46794db-2c88-4421-89b1-3fd6ce8b2ade None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:38.806662 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-c46794db-2c88-4421-89b1-3fd6ce8b2ade None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:45:38.807992 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-c46794db-2c88-4421-89b1-3fd6ce8b2ade None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:38.917792 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:45:38.926060 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 684/1381] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:45:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:45:39.240699 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1187ba7f-7ad9-4eb1-867a-41bdbb9c9e93 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:45:39.242813 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-1187ba7f-7ad9-4eb1-867a-41bdbb9c9e93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:39.395740 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:45:39.407138 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1187ba7f-7ad9-4eb1-867a-41bdbb9c9e93 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:45:39.541236 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-294f7563-5eb4-4813-82b9-424e1909fdf1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '835def0e6c6d4365a2767051509eb0d6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'ResellerAdmin', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:45:39.541899 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294f7563-5eb4-4813-82b9-424e1909fdf1 service nova] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:39.542175 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294f7563-5eb4-4813-82b9-424e1909fdf1 service nova] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:39.542443 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294f7563-5eb4-4813-82b9-424e1909fdf1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:39.544100 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-294f7563-5eb4-4813-82b9-424e1909fdf1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:39.647432 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-294f7563-5eb4-4813-82b9-424e1909fdf1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:45:39.649056 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-294f7563-5eb4-4813-82b9-424e1909fdf1 service nova] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:45:39.659334 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:45:39.669056 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 698/1382] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:45:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 430 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:45:41.910819 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00f474c-fb8f-47cb-9ac4-d561b4362b74 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:41.911493 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00f474c-fb8f-47cb-9ac4-d561b4362b74 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:41.911493 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00f474c-fb8f-47cb-9ac4-d561b4362b74 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:41.913501 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 685/1383] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:45:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:45:41.919169 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d10f7ef-e434-4453-82cf-c76cc75a4498 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:41.919610 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d10f7ef-e434-4453-82cf-c76cc75a4498 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:41.919703 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d10f7ef-e434-4453-82cf-c76cc75a4498 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:41.995470 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-8d10f7ef-e434-4453-82cf-c76cc75a4498 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:45:41.996697 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-8d10f7ef-e434-4453-82cf-c76cc75a4498 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:42.106315 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 699/1384] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:45:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:45:42.406969 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c00f474c-fb8f-47cb-9ac4-d561b4362b74 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:45:42.408496 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-c00f474c-fb8f-47cb-9ac4-d561b4362b74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:42.566111 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c00f474c-fb8f-47cb-9ac4-d561b4362b74 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:45:42.642577 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:45:42.703797 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a00a6a36-d67e-4812-b92d-0db218580783 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '835def0e6c6d4365a2767051509eb0d6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'ResellerAdmin', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:45:42.704585 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a00a6a36-d67e-4812-b92d-0db218580783 service nova] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:42.704939 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a00a6a36-d67e-4812-b92d-0db218580783 service nova] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:42.705292 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a00a6a36-d67e-4812-b92d-0db218580783 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:42.707507 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-a00a6a36-d67e-4812-b92d-0db218580783 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:42.808503 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a00a6a36-d67e-4812-b92d-0db218580783 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:45:42.810101 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a00a6a36-d67e-4812-b92d-0db218580783 service nova] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:45:42.831044 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 686/1385] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:45:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 425 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:45:45.022113 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a48bc99f-9f08-4d17-8fca-34fd9890de25 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:45.022726 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a48bc99f-9f08-4d17-8fca-34fd9890de25 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:45.022726 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a48bc99f-9f08-4d17-8fca-34fd9890de25 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:45.024066 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 700/1386] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:45:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:45:45.029344 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5e3484-80cc-4857-95ef-92d98a4372eb None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:45.029765 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5e3484-80cc-4857-95ef-92d98a4372eb None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:45.030015 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5e3484-80cc-4857-95ef-92d98a4372eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:45.102804 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-fb5e3484-80cc-4857-95ef-92d98a4372eb None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:45:45.104041 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-fb5e3484-80cc-4857-95ef-92d98a4372eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:45.137727 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:45:45.203644 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 687/1387] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:45:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:45:45.497248 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a48bc99f-9f08-4d17-8fca-34fd9890de25 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:45:45.498899 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-a48bc99f-9f08-4d17-8fca-34fd9890de25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:45.578965 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:45:45.647043 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a48bc99f-9f08-4d17-8fca-34fd9890de25 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:45:45.759287 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41830f4d-5266-4a0a-934e-bedef95c942f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '835def0e6c6d4365a2767051509eb0d6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'ResellerAdmin', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:45:45.759812 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41830f4d-5266-4a0a-934e-bedef95c942f service nova] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:45.760088 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41830f4d-5266-4a0a-934e-bedef95c942f service nova] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:45.760358 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41830f4d-5266-4a0a-934e-bedef95c942f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:45.761930 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-41830f4d-5266-4a0a-934e-bedef95c942f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:45.801651 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:45:45.855306 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41830f4d-5266-4a0a-934e-bedef95c942f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:45:45.856922 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41830f4d-5266-4a0a-934e-bedef95c942f service nova] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:45:45.876038 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 701/1388] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:45:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 379 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:45:46.783525 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcddcdbd-e7a3-4f54-9787-7509b88104e4 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:46.784082 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcddcdbd-e7a3-4f54-9787-7509b88104e4 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:46.784082 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcddcdbd-e7a3-4f54-9787-7509b88104e4 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:46.785656 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 688/1389] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:45:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:45:46.792382 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb68ea1-5ae0-410d-9143-fc5a76226d52 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:46.792971 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb68ea1-5ae0-410d-9143-fc5a76226d52 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:46.792971 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb68ea1-5ae0-410d-9143-fc5a76226d52 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:46.862036 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-8eb68ea1-5ae0-410d-9143-fc5a76226d52 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:45:46.863287 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-8eb68ea1-5ae0-410d-9143-fc5a76226d52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:46.963678 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 702/1390] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:45:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:45:47.272198 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c68ff08-b3d5-4fd3-aabd-de1066f2906c None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:47.272548 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c68ff08-b3d5-4fd3-aabd-de1066f2906c None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:47.272890 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c68ff08-b3d5-4fd3-aabd-de1066f2906c None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:47.274450 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 689/1391] 10.222.0.27 () {54 vars in 966 bytes} [Sun Sep 11 00:45:47 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:45:47.279581 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8999c9-fbe2-4891-9440-982c253d67a6 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:47.279865 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8999c9-fbe2-4891-9440-982c253d67a6 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:47.280135 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8999c9-fbe2-4891-9440-982c253d67a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:47.297513 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:45:47.360282 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-7c8999c9-fbe2-4891-9440-982c253d67a6 None None] MFA Rules not processed for user `06b1c911731f4267a5f37e91d948e7e9`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:45:47.361472 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-7c8999c9-fbe2-4891-9440-982c253d67a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:47.437648 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 703/1392] 10.222.0.27 () {58 vars in 1063 bytes} [Sun Sep 11 00:45:47 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:45:47.446093 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c68ff08-b3d5-4fd3-aabd-de1066f2906c None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:45:47.447732 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-0c68ff08-b3d5-4fd3-aabd-de1066f2906c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:47.553809 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c68ff08-b3d5-4fd3-aabd-de1066f2906c None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:45:47.611931 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:45:47.627897 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6411a9a-22b8-44c6-9442-23b5f699641e service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '06b1c911731f4267a5f37e91d948e7e9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:45:47.628590 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6411a9a-22b8-44c6-9442-23b5f699641e service glance] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:47.628863 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6411a9a-22b8-44c6-9442-23b5f699641e service glance] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:47.629124 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6411a9a-22b8-44c6-9442-23b5f699641e service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:47.631515 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-f6411a9a-22b8-44c6-9442-23b5f699641e service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:47.743332 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6411a9a-22b8-44c6-9442-23b5f699641e service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:45:47.744794 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6411a9a-22b8-44c6-9442-23b5f699641e service glance] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:45:47.764221 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 690/1393] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:45:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:45:48.091911 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3396ab68-1d35-4d9e-ba67-d52b1c066f0e None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:48.092441 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3396ab68-1d35-4d9e-ba67-d52b1c066f0e None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:48.092441 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3396ab68-1d35-4d9e-ba67-d52b1c066f0e None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:48.093820 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 704/1394] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:45:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:45:48.100430 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08657298-f9cc-4fb0-bbdc-cd78313282c5 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:48.101002 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08657298-f9cc-4fb0-bbdc-cd78313282c5 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:48.101002 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08657298-f9cc-4fb0-bbdc-cd78313282c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:48.170424 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-08657298-f9cc-4fb0-bbdc-cd78313282c5 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:45:48.171663 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-08657298-f9cc-4fb0-bbdc-cd78313282c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:48.216288 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:45:48.273672 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 691/1395] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:45:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:45:48.526446 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e50396-6f01-4386-8af9-788770b7409a None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:48.526732 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e50396-6f01-4386-8af9-788770b7409a None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:48.527050 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e50396-6f01-4386-8af9-788770b7409a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:48.572691 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08657298-f9cc-4fb0-bbdc-cd78313282c5 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:45:48.573895 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-08657298-f9cc-4fb0-bbdc-cd78313282c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:48.598078 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-89e50396-6f01-4386-8af9-788770b7409a None None] MFA Rules not processed for user `9fc4274836844ae4ad5d16d74c36f671`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:45:48.599392 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-89e50396-6f01-4386-8af9-788770b7409a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:48.654630 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:45:48.686502 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 705/1396] 10.222.0.27 () {58 vars in 1009 bytes} [Sun Sep 11 00:45:48 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:45:48.691380 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08657298-f9cc-4fb0-bbdc-cd78313282c5 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:45:48.734498 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11bb2637-2476-4aa3-a23a-401bbbb47d44 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:48.734817 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11bb2637-2476-4aa3-a23a-401bbbb47d44 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:48.735090 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11bb2637-2476-4aa3-a23a-401bbbb47d44 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:48.736442 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 706/1397] 10.222.0.27 () {54 vars in 943 bytes} [Sun Sep 11 00:45:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 11 00:45:48.742994 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f41746-ec24-49b0-a515-7d48abee71ae None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:48.743283 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f41746-ec24-49b0-a515-7d48abee71ae None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:48.743563 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f41746-ec24-49b0-a515-7d48abee71ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:48.753800 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:45:48.815354 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cd2ec1e-43cf-4917-ac21-0947a92c3d7d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '835def0e6c6d4365a2767051509eb0d6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'ResellerAdmin', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:45:48.815991 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd2ec1e-43cf-4917-ac21-0947a92c3d7d service nova] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:48.816268 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd2ec1e-43cf-4917-ac21-0947a92c3d7d service nova] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:48.816536 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd2ec1e-43cf-4917-ac21-0947a92c3d7d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:48.818184 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-8cd2ec1e-43cf-4917-ac21-0947a92c3d7d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:48.822215 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-40f41746-ec24-49b0-a515-7d48abee71ae None None] MFA Rules not processed for user `bcb73dec73464a3abc4b774abb4f416f`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:45:48.823426 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-40f41746-ec24-49b0-a515-7d48abee71ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:48.912774 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cd2ec1e-43cf-4917-ac21-0947a92c3d7d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:45:48.914550 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cd2ec1e-43cf-4917-ac21-0947a92c3d7d service nova] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:45:48.932671 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 692/1398] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:45:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 360 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 11 00:45:48.939707 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 707/1399] 10.222.0.27 () {58 vars in 1040 bytes} [Sun Sep 11 00:45:48 2022] POST /identity/v3/auth/tokens => generated 3304 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:45:48.949861 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cd2ec1e-43cf-4917-ac21-0947a92c3d7d service nova] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:45:48.951095 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-8cd2ec1e-43cf-4917-ac21-0947a92c3d7d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:49.024912 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:45:49.055319 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cd2ec1e-43cf-4917-ac21-0947a92c3d7d service nova] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:45:49.174599 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc5fc95-f6f2-4e63-bb48-145d32835e91 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bcb73dec73464a3abc4b774abb4f416f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:45:49.175379 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc5fc95-f6f2-4e63-bb48-145d32835e91 service swift] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:49.175555 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc5fc95-f6f2-4e63-bb48-145d32835e91 service swift] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:49.175819 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc5fc95-f6f2-4e63-bb48-145d32835e91 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:49.177566 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-8cc5fc95-f6f2-4e63-bb48-145d32835e91 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:49.262675 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc5fc95-f6f2-4e63-bb48-145d32835e91 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:45:49.264229 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc5fc95-f6f2-4e63-bb48-145d32835e91 service swift] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:45:49.272386 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 693/1400] 10.222.0.27 () {60 vars in 1384 bytes} [Sun Sep 11 00:45:48 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 11 00:45:51.163995 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ddc9e9-3d02-43d3-ac60-024e263b5845 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:51.164552 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ddc9e9-3d02-43d3-ac60-024e263b5845 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:51.164552 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ddc9e9-3d02-43d3-ac60-024e263b5845 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:51.166062 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 708/1401] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:45:51 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:45:51.171299 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-025dbaec-2b26-4e1f-8162-628fe209570f None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:51.171812 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-025dbaec-2b26-4e1f-8162-628fe209570f None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:51.171812 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-025dbaec-2b26-4e1f-8162-628fe209570f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:51.187000 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:45:51.246019 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-025dbaec-2b26-4e1f-8162-628fe209570f None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:45:51.247369 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-025dbaec-2b26-4e1f-8162-628fe209570f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:51.372941 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 694/1402] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:45:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:45:51.681589 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ddc9e9-3d02-43d3-ac60-024e263b5845 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:45:51.683207 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-b4ddc9e9-3d02-43d3-ac60-024e263b5845 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:51.716524 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:45:51.845709 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ddc9e9-3d02-43d3-ac60-024e263b5845 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:45:51.961416 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:45:51.975051 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db84a4ce-e7b9-48af-995d-962e24598b47 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '835def0e6c6d4365a2767051509eb0d6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'ResellerAdmin', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:45:51.976063 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db84a4ce-e7b9-48af-995d-962e24598b47 service nova] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:51.976501 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db84a4ce-e7b9-48af-995d-962e24598b47 service nova] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:51.976795 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db84a4ce-e7b9-48af-995d-962e24598b47 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:51.978477 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-db84a4ce-e7b9-48af-995d-962e24598b47 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:52.083261 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db84a4ce-e7b9-48af-995d-962e24598b47 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:45:52.085781 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db84a4ce-e7b9-48af-995d-962e24598b47 service nova] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:45:52.108039 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 709/1403] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:45:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 427 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:45:54.355755 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81f5c25-ed5c-4a82-aec9-1b9027ee6b97 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:54.356336 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81f5c25-ed5c-4a82-aec9-1b9027ee6b97 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:54.356336 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81f5c25-ed5c-4a82-aec9-1b9027ee6b97 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:54.357895 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 695/1404] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:45:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:45:54.362948 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd0806db-79a5-41b1-84e8-1c4720d9aa29 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:54.363399 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd0806db-79a5-41b1-84e8-1c4720d9aa29 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:54.363574 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd0806db-79a5-41b1-84e8-1c4720d9aa29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:54.442520 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-dd0806db-79a5-41b1-84e8-1c4720d9aa29 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:45:54.444487 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-dd0806db-79a5-41b1-84e8-1c4720d9aa29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:54.523818 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:45:54.568139 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 710/1405] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:45:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:45:54.862310 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e81f5c25-ed5c-4a82-aec9-1b9027ee6b97 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:45:54.863873 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-e81f5c25-ed5c-4a82-aec9-1b9027ee6b97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:54.966129 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:45:55.030333 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e81f5c25-ed5c-4a82-aec9-1b9027ee6b97 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:45:55.163414 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18139bf-4e85-405b-8541-7b16b91affeb service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '835def0e6c6d4365a2767051509eb0d6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'ResellerAdmin', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:45:55.164209 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18139bf-4e85-405b-8541-7b16b91affeb service nova] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:55.164478 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18139bf-4e85-405b-8541-7b16b91affeb service nova] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:55.164743 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18139bf-4e85-405b-8541-7b16b91affeb service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:55.166349 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-f18139bf-4e85-405b-8541-7b16b91affeb service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:55.225359 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:45:55.269054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f18139bf-4e85-405b-8541-7b16b91affeb service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:45:55.270819 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f18139bf-4e85-405b-8541-7b16b91affeb service nova] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:45:55.290895 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 696/1406] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:45:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 429 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:45:57.584859 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3195cfc-36ab-46a1-9ffd-b103670525f3 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:57.584859 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3195cfc-36ab-46a1-9ffd-b103670525f3 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:57.584859 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3195cfc-36ab-46a1-9ffd-b103670525f3 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:57.586996 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 711/1407] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:45:57 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:45:57.591477 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0a56c6-3d9c-4cf2-a23e-b82dc8675dd7 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:57.591913 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0a56c6-3d9c-4cf2-a23e-b82dc8675dd7 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:57.592017 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0a56c6-3d9c-4cf2-a23e-b82dc8675dd7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:57.664974 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-7e0a56c6-3d9c-4cf2-a23e-b82dc8675dd7 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:45:57.666443 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-7e0a56c6-3d9c-4cf2-a23e-b82dc8675dd7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:57.776523 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 697/1408] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:45:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:45:58.069155 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3195cfc-36ab-46a1-9ffd-b103670525f3 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:45:58.070696 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-d3195cfc-36ab-46a1-9ffd-b103670525f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:58.237325 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3195cfc-36ab-46a1-9ffd-b103670525f3 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:45:58.292062 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:45:58.362810 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288284b1-ed3b-4c7e-ae98-ebab4b9854e0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '835def0e6c6d4365a2767051509eb0d6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'ResellerAdmin', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:45:58.363349 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288284b1-ed3b-4c7e-ae98-ebab4b9854e0 service nova] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:45:58.363631 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288284b1-ed3b-4c7e-ae98-ebab4b9854e0 service nova] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:45:58.363901 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288284b1-ed3b-4c7e-ae98-ebab4b9854e0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:45:58.365573 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-288284b1-ed3b-4c7e-ae98-ebab4b9854e0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:45:58.464509 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-288284b1-ed3b-4c7e-ae98-ebab4b9854e0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:45:58.469567 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-288284b1-ed3b-4c7e-ae98-ebab4b9854e0 service nova] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:45:58.491021 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 712/1409] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:45:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 423 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:46:07.774401 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=87 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:46:08.487235 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=76 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:46:12.316557 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192b4825-e065-41e5-ad31-88a8c88d8e10 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:12.317145 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192b4825-e065-41e5-ad31-88a8c88d8e10 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:12.317248 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192b4825-e065-41e5-ad31-88a8c88d8e10 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:12.318614 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 698/1410] 10.222.0.27 () {54 vars in 918 bytes} [Sun Sep 11 00:46:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:46:12.323783 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080ab56b-efba-4df4-8500-bfcb5b0d2dab None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:12.324204 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080ab56b-efba-4df4-8500-bfcb5b0d2dab None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:12.324320 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080ab56b-efba-4df4-8500-bfcb5b0d2dab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:12.399792 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-080ab56b-efba-4df4-8500-bfcb5b0d2dab None None] MFA Rules not processed for user `cdbfc2ccd73e4a5eb8c999be5b094619`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:46:12.400900 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-080ab56b-efba-4df4-8500-bfcb5b0d2dab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:12.508662 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 713/1411] 10.222.0.27 () {58 vars in 1015 bytes} [Sun Sep 11 00:46:12 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:46:12.523355 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67304c67-50fc-46cb-97e1-e8aa58238c93 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:12.523741 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67304c67-50fc-46cb-97e1-e8aa58238c93 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:12.524116 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67304c67-50fc-46cb-97e1-e8aa58238c93 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:12.525537 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 699/1412] 10.222.0.27 () {54 vars in 918 bytes} [Sun Sep 11 00:46:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:46:12.530085 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e4e409-eec1-4059-aa03-7df060a1044e None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:12.530370 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e4e409-eec1-4059-aa03-7df060a1044e None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:12.530643 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e4e409-eec1-4059-aa03-7df060a1044e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:12.602307 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:46:12.605439 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-e0e4e409-eec1-4059-aa03-7df060a1044e None None] MFA Rules not processed for user `835def0e6c6d4365a2767051509eb0d6`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:46:12.607186 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-e0e4e409-eec1-4059-aa03-7df060a1044e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:12.721476 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 714/1413] 10.222.0.27 () {58 vars in 1015 bytes} [Sun Sep 11 00:46:12 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:46:15.307212 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-374d74ec-08a6-48e7-a9d1-6cce54f13ea8 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:15.307751 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-374d74ec-08a6-48e7-a9d1-6cce54f13ea8 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:15.307751 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-374d74ec-08a6-48e7-a9d1-6cce54f13ea8 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:15.309109 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 700/1414] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 11 00:46:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:46:15.314556 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf904f6-ac3d-4d77-9946-257f4986689f None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:15.315044 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf904f6-ac3d-4d77-9946-257f4986689f None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:15.315150 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf904f6-ac3d-4d77-9946-257f4986689f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:15.380439 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-fcf904f6-ac3d-4d77-9946-257f4986689f None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:46:15.381530 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-fcf904f6-ac3d-4d77-9946-257f4986689f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:15.436863 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:46:15.480013 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 715/1415] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 11 00:46:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:46:15.770186 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-374d74ec-08a6-48e7-a9d1-6cce54f13ea8 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:46:15.771937 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-374d74ec-08a6-48e7-a9d1-6cce54f13ea8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:15.916556 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-374d74ec-08a6-48e7-a9d1-6cce54f13ea8 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:46:15.988455 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:46:16.029140 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc8d1c3-2dd4-43f8-95c1-420ec965b0eb service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '835def0e6c6d4365a2767051509eb0d6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'ResellerAdmin', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:46:16.029669 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc8d1c3-2dd4-43f8-95c1-420ec965b0eb service nova] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:16.029945 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc8d1c3-2dd4-43f8-95c1-420ec965b0eb service nova] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:16.030208 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc8d1c3-2dd4-43f8-95c1-420ec965b0eb service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:16.031812 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-5dc8d1c3-2dd4-43f8-95c1-420ec965b0eb service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:16.121076 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dc8d1c3-2dd4-43f8-95c1-420ec965b0eb service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:46:16.122670 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dc8d1c3-2dd4-43f8-95c1-420ec965b0eb service nova] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:46:16.141345 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 701/1416] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:46:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:46:17.287787 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d1d348-efe5-48f9-9dd4-dbcfcfaadf2a None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:17.288355 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d1d348-efe5-48f9-9dd4-dbcfcfaadf2a None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:17.288355 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d1d348-efe5-48f9-9dd4-dbcfcfaadf2a None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:17.289703 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 716/1417] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 11 00:46:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:46:17.295858 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a386aa3b-8c0a-4032-afa9-92a8eb2b84ff None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:17.296329 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a386aa3b-8c0a-4032-afa9-92a8eb2b84ff None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:17.296428 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a386aa3b-8c0a-4032-afa9-92a8eb2b84ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:17.373478 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-a386aa3b-8c0a-4032-afa9-92a8eb2b84ff None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:46:17.374593 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-a386aa3b-8c0a-4032-afa9-92a8eb2b84ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:17.401786 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:46:17.471629 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 702/1418] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 11 00:46:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:46:17.623835 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d1d348-efe5-48f9-9dd4-dbcfcfaadf2a None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:46:17.625288 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-43d1d348-efe5-48f9-9dd4-dbcfcfaadf2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:17.772404 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d1d348-efe5-48f9-9dd4-dbcfcfaadf2a None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:46:17.814622 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:46:17.888198 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd2506b-0fac-4457-9ff2-2cbda470b37d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '835def0e6c6d4365a2767051509eb0d6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'ResellerAdmin', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:46:17.888749 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd2506b-0fac-4457-9ff2-2cbda470b37d service nova] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:17.889078 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd2506b-0fac-4457-9ff2-2cbda470b37d service nova] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:17.889379 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd2506b-0fac-4457-9ff2-2cbda470b37d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:17.890978 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-bcd2506b-0fac-4457-9ff2-2cbda470b37d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:17.980425 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcd2506b-0fac-4457-9ff2-2cbda470b37d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:46:17.982033 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcd2506b-0fac-4457-9ff2-2cbda470b37d service nova] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:46:18.000168 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 717/1419] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:46:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 377 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:46:19.133778 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210443e3-f730-4d85-83e2-1d574316957d None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:19.134202 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210443e3-f730-4d85-83e2-1d574316957d None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:19.134326 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210443e3-f730-4d85-83e2-1d574316957d None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:19.135697 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 703/1420] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 11 00:46:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:46:19.141960 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e36933-7b20-43b0-913a-09b5fb798a49 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:19.142437 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e36933-7b20-43b0-913a-09b5fb798a49 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:19.142545 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e36933-7b20-43b0-913a-09b5fb798a49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:19.197677 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:46:19.219473 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-37e36933-7b20-43b0-913a-09b5fb798a49 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:46:19.220669 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-37e36933-7b20-43b0-913a-09b5fb798a49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:19.316146 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 718/1421] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 11 00:46:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:46:19.468266 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-210443e3-f730-4d85-83e2-1d574316957d None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:46:19.469828 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-210443e3-f730-4d85-83e2-1d574316957d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:19.617917 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-210443e3-f730-4d85-83e2-1d574316957d None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:46:19.629653 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:46:19.731753 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e36c84c4-b812-48de-b962-f1c044be018f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '835def0e6c6d4365a2767051509eb0d6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'ResellerAdmin', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:46:19.732276 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36c84c4-b812-48de-b962-f1c044be018f service nova] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:19.732553 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36c84c4-b812-48de-b962-f1c044be018f service nova] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:19.732821 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36c84c4-b812-48de-b962-f1c044be018f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:19.734403 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-e36c84c4-b812-48de-b962-f1c044be018f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:19.822821 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e36c84c4-b812-48de-b962-f1c044be018f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:46:19.824429 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e36c84c4-b812-48de-b962-f1c044be018f service nova] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:46:19.842622 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 704/1422] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:46:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 375 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:46:20.952417 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823b6ca3-398c-42ec-8e23-c0244fb7308b None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:20.952942 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823b6ca3-398c-42ec-8e23-c0244fb7308b None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:20.952942 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823b6ca3-398c-42ec-8e23-c0244fb7308b None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:20.954334 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 719/1423] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 11 00:46:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:46:20.961158 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0758a0-5acb-4b33-9ffe-c7a92eccaa7a None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:20.961613 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0758a0-5acb-4b33-9ffe-c7a92eccaa7a None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:20.961725 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0758a0-5acb-4b33-9ffe-c7a92eccaa7a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:20.982136 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:46:21.039673 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-ec0758a0-5acb-4b33-9ffe-c7a92eccaa7a None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:46:21.040873 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-ec0758a0-5acb-4b33-9ffe-c7a92eccaa7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:21.139245 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 705/1424] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 11 00:46:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:46:21.290495 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-823b6ca3-398c-42ec-8e23-c0244fb7308b None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:46:21.292075 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-823b6ca3-398c-42ec-8e23-c0244fb7308b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:21.400745 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:46:21.436988 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-823b6ca3-398c-42ec-8e23-c0244fb7308b None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:46:21.548484 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d2e1ee7-cbba-4186-b232-fecc36d13357 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '835def0e6c6d4365a2767051509eb0d6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'ResellerAdmin', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:46:21.549085 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d2e1ee7-cbba-4186-b232-fecc36d13357 service nova] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:21.549363 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d2e1ee7-cbba-4186-b232-fecc36d13357 service nova] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:21.549631 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d2e1ee7-cbba-4186-b232-fecc36d13357 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:21.551235 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-6d2e1ee7-cbba-4186-b232-fecc36d13357 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:21.625727 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:46:21.639889 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d2e1ee7-cbba-4186-b232-fecc36d13357 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:46:21.641471 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d2e1ee7-cbba-4186-b232-fecc36d13357 service nova] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:46:21.660023 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 720/1425] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:46:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:46:22.783833 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872c7da0-74da-4d23-a62f-746986c822cb None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:22.784466 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872c7da0-74da-4d23-a62f-746986c822cb None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:22.784466 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872c7da0-74da-4d23-a62f-746986c822cb None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:22.785754 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 706/1426] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 11 00:46:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:46:22.792101 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0419741-fc5f-4f9f-a9d6-99c044cb7513 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:22.792541 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0419741-fc5f-4f9f-a9d6-99c044cb7513 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:22.792713 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0419741-fc5f-4f9f-a9d6-99c044cb7513 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:22.868708 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-e0419741-fc5f-4f9f-a9d6-99c044cb7513 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:46:22.869808 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-e0419741-fc5f-4f9f-a9d6-99c044cb7513 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:22.965380 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 721/1427] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 11 00:46:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:46:23.116468 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-872c7da0-74da-4d23-a62f-746986c822cb None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:46:23.117954 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-872c7da0-74da-4d23-a62f-746986c822cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:23.202449 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:46:23.265047 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-872c7da0-74da-4d23-a62f-746986c822cb None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:46:23.376785 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71363103-9c21-4a9d-a451-87c783e444f0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '835def0e6c6d4365a2767051509eb0d6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'ResellerAdmin', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:46:23.377412 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71363103-9c21-4a9d-a451-87c783e444f0 service nova] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:23.377687 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71363103-9c21-4a9d-a451-87c783e444f0 service nova] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:23.377958 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71363103-9c21-4a9d-a451-87c783e444f0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:23.379565 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-71363103-9c21-4a9d-a451-87c783e444f0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:23.428734 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:46:23.469233 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71363103-9c21-4a9d-a451-87c783e444f0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:46:23.470857 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71363103-9c21-4a9d-a451-87c783e444f0 service nova] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:46:23.488861 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 707/1428] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:46:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:46:24.546179 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a57991-7640-4f64-a2d2-4d55617726f3 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:24.546727 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a57991-7640-4f64-a2d2-4d55617726f3 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:24.546727 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a57991-7640-4f64-a2d2-4d55617726f3 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:24.548079 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 722/1429] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 11 00:46:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:46:24.554992 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf2ad9f-c1da-4d86-a038-7ef82020d9cc None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:24.555438 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf2ad9f-c1da-4d86-a038-7ef82020d9cc None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:24.555540 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf2ad9f-c1da-4d86-a038-7ef82020d9cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:24.632990 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-7bf2ad9f-c1da-4d86-a038-7ef82020d9cc None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:46:24.634185 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-7bf2ad9f-c1da-4d86-a038-7ef82020d9cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:24.734808 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 708/1430] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 11 00:46:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:46:24.885882 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a57991-7640-4f64-a2d2-4d55617726f3 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:46:24.887314 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-a9a57991-7640-4f64-a2d2-4d55617726f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:25.051108 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:46:25.147714 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a57991-7640-4f64-a2d2-4d55617726f3 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:46:25.258663 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7005a349-6b90-43e5-ac13-aeb5fe6f6917 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '835def0e6c6d4365a2767051509eb0d6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'ResellerAdmin', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:46:25.259299 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7005a349-6b90-43e5-ac13-aeb5fe6f6917 service nova] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:25.259572 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7005a349-6b90-43e5-ac13-aeb5fe6f6917 service nova] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:25.259833 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7005a349-6b90-43e5-ac13-aeb5fe6f6917 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:25.261402 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-7005a349-6b90-43e5-ac13-aeb5fe6f6917 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:25.268539 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:46:25.353097 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7005a349-6b90-43e5-ac13-aeb5fe6f6917 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:46:25.354683 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7005a349-6b90-43e5-ac13-aeb5fe6f6917 service nova] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:46:25.372278 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 723/1431] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:46:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 487 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:46:26.435045 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf5e986-9540-45df-b9a5-c1cfdaef7b7e None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:26.435577 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf5e986-9540-45df-b9a5-c1cfdaef7b7e None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:26.435577 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf5e986-9540-45df-b9a5-c1cfdaef7b7e None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:26.436831 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 709/1432] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 11 00:46:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:46:26.443568 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e7a820-08d1-4183-8c7c-9d0bff938db8 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:26.444106 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e7a820-08d1-4183-8c7c-9d0bff938db8 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:26.444106 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e7a820-08d1-4183-8c7c-9d0bff938db8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:26.518740 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-d4e7a820-08d1-4183-8c7c-9d0bff938db8 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:46:26.519967 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-d4e7a820-08d1-4183-8c7c-9d0bff938db8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:26.589798 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:46:26.618304 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 724/1433] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 11 00:46:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:46:26.769511 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf5e986-9540-45df-b9a5-c1cfdaef7b7e None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:46:26.771694 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-acf5e986-9540-45df-b9a5-c1cfdaef7b7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:26.789011 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:46:26.922637 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf5e986-9540-45df-b9a5-c1cfdaef7b7e None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:46:27.009639 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:46:27.035802 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-139454f8-91a9-4615-bbdb-1acc530f0787 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '835def0e6c6d4365a2767051509eb0d6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'ResellerAdmin', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:46:27.036354 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139454f8-91a9-4615-bbdb-1acc530f0787 service nova] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:27.036632 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139454f8-91a9-4615-bbdb-1acc530f0787 service nova] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:27.036908 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139454f8-91a9-4615-bbdb-1acc530f0787 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:27.038477 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-139454f8-91a9-4615-bbdb-1acc530f0787 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:27.128505 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-139454f8-91a9-4615-bbdb-1acc530f0787 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:46:27.130181 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-139454f8-91a9-4615-bbdb-1acc530f0787 service nova] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:46:27.149673 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 710/1434] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:46:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 381 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:46:28.224503 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b0f4c45-25fa-46fb-a45f-a72d636a4628 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:28.225066 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b0f4c45-25fa-46fb-a45f-a72d636a4628 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:28.225306 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b0f4c45-25fa-46fb-a45f-a72d636a4628 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:28.226648 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 725/1435] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 11 00:46:28 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:46:28.232815 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bdc938d-68eb-45f9-b332-0c5ddd08fedf None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:28.233296 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bdc938d-68eb-45f9-b332-0c5ddd08fedf None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:28.233406 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bdc938d-68eb-45f9-b332-0c5ddd08fedf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:28.309120 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-8bdc938d-68eb-45f9-b332-0c5ddd08fedf None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:46:28.310310 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-8bdc938d-68eb-45f9-b332-0c5ddd08fedf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:28.352427 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:46:28.407881 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 711/1436] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 11 00:46:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:46:28.559183 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b0f4c45-25fa-46fb-a45f-a72d636a4628 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:46:28.560685 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-1b0f4c45-25fa-46fb-a45f-a72d636a4628 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:28.704794 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b0f4c45-25fa-46fb-a45f-a72d636a4628 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:46:28.758264 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:46:28.815661 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a11a0d2a-c12d-4c5a-ad32-08cfea85f68c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '835def0e6c6d4365a2767051509eb0d6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'ResellerAdmin', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:46:28.816268 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11a0d2a-c12d-4c5a-ad32-08cfea85f68c service nova] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:28.816546 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11a0d2a-c12d-4c5a-ad32-08cfea85f68c service nova] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:28.816814 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11a0d2a-c12d-4c5a-ad32-08cfea85f68c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:28.819026 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-a11a0d2a-c12d-4c5a-ad32-08cfea85f68c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:28.907018 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a11a0d2a-c12d-4c5a-ad32-08cfea85f68c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:46:28.908864 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a11a0d2a-c12d-4c5a-ad32-08cfea85f68c service nova] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:46:28.927594 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 726/1437] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:46:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 368 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:46:29.983768 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe62afd-df9f-4bfb-8b3c-fcaf91320768 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:29.984329 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe62afd-df9f-4bfb-8b3c-fcaf91320768 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:29.984329 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe62afd-df9f-4bfb-8b3c-fcaf91320768 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:29.985661 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 712/1438] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 11 00:46:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:46:29.992071 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec4b07c-8931-4667-9da6-011d68c7dc4f None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:29.992526 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec4b07c-8931-4667-9da6-011d68c7dc4f None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:29.992635 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec4b07c-8931-4667-9da6-011d68c7dc4f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:30.072112 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-3ec4b07c-8931-4667-9da6-011d68c7dc4f None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:46:30.073300 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-3ec4b07c-8931-4667-9da6-011d68c7dc4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:30.075301 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:46:30.171671 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 727/1439] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 11 00:46:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:46:30.335092 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afe62afd-df9f-4bfb-8b3c-fcaf91320768 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:46:30.336499 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-afe62afd-df9f-4bfb-8b3c-fcaf91320768 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:30.481265 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afe62afd-df9f-4bfb-8b3c-fcaf91320768 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:46:30.508745 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:46:30.594336 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87616e11-ab61-43a7-a878-ff830b1b8aa7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '835def0e6c6d4365a2767051509eb0d6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'ResellerAdmin', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:46:30.594918 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87616e11-ab61-43a7-a878-ff830b1b8aa7 service nova] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:30.595191 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87616e11-ab61-43a7-a878-ff830b1b8aa7 service nova] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:30.595456 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87616e11-ab61-43a7-a878-ff830b1b8aa7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:30.597080 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-87616e11-ab61-43a7-a878-ff830b1b8aa7 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:30.686551 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87616e11-ab61-43a7-a878-ff830b1b8aa7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:46:30.688193 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87616e11-ab61-43a7-a878-ff830b1b8aa7 service nova] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:46:30.706103 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 713/1440] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:46:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:46:32.775880 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2640ac91-158d-4045-beca-c4e5920d9a31 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:32.776475 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2640ac91-158d-4045-beca-c4e5920d9a31 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:32.776475 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2640ac91-158d-4045-beca-c4e5920d9a31 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:32.777834 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 728/1441] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 11 00:46:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:46:32.783341 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d05b198a-8112-4950-aed2-cbf85c70663d None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:32.783736 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d05b198a-8112-4950-aed2-cbf85c70663d None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:32.783887 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d05b198a-8112-4950-aed2-cbf85c70663d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:32.824008 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:46:32.849913 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-d05b198a-8112-4950-aed2-cbf85c70663d None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:46:32.851219 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-d05b198a-8112-4950-aed2-cbf85c70663d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:32.949697 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 714/1442] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 11 00:46:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:46:33.238281 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2640ac91-158d-4045-beca-c4e5920d9a31 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:46:33.240558 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-2640ac91-158d-4045-beca-c4e5920d9a31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:33.342888 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2640ac91-158d-4045-beca-c4e5920d9a31 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:46:33.374915 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:46:33.417364 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a221309-b113-48fd-95e0-fd2795df5c29 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:46:33.417961 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a221309-b113-48fd-95e0-fd2795df5c29 service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:33.418234 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a221309-b113-48fd-95e0-fd2795df5c29 service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:33.418497 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a221309-b113-48fd-95e0-fd2795df5c29 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:33.420086 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-2a221309-b113-48fd-95e0-fd2795df5c29 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:33.527644 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a221309-b113-48fd-95e0-fd2795df5c29 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:46:33.529066 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a221309-b113-48fd-95e0-fd2795df5c29 service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:46:33.547980 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 729/1443] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:46:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:46:36.072496 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1223de-b2b9-4d2e-982b-77de8c22af38 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:36.073125 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1223de-b2b9-4d2e-982b-77de8c22af38 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:36.073125 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1223de-b2b9-4d2e-982b-77de8c22af38 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:36.074417 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 715/1444] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 11 00:46:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:46:36.080218 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d52d86e9-00fd-424e-a1ce-b1baa7db0b40 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:36.081020 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d52d86e9-00fd-424e-a1ce-b1baa7db0b40 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:36.081020 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d52d86e9-00fd-424e-a1ce-b1baa7db0b40 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:36.151706 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-d52d86e9-00fd-424e-a1ce-b1baa7db0b40 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:46:36.152914 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-d52d86e9-00fd-424e-a1ce-b1baa7db0b40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:36.165817 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:46:36.249879 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 730/1445] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 11 00:46:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:46:36.538648 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d1223de-b2b9-4d2e-982b-77de8c22af38 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:46:36.540264 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-7d1223de-b2b9-4d2e-982b-77de8c22af38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:36.642610 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d1223de-b2b9-4d2e-982b-77de8c22af38 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:46:36.654560 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:46:36.717993 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dde01061-806d-419e-a2e1-835abc67a6dd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:46:36.719061 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde01061-806d-419e-a2e1-835abc67a6dd service neutron] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:36.719551 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde01061-806d-419e-a2e1-835abc67a6dd service neutron] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:36.720083 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde01061-806d-419e-a2e1-835abc67a6dd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:36.723135 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-dde01061-806d-419e-a2e1-835abc67a6dd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:36.837196 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dde01061-806d-419e-a2e1-835abc67a6dd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:46:36.838552 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dde01061-806d-419e-a2e1-835abc67a6dd service neutron] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:46:36.856459 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 716/1446] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:46:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:46:39.315176 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38b5db7-c321-45c0-ae55-e1f8c6d6ac48 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:39.315750 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38b5db7-c321-45c0-ae55-e1f8c6d6ac48 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:39.315750 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38b5db7-c321-45c0-ae55-e1f8c6d6ac48 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:39.317068 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 731/1447] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 11 00:46:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 11 00:46:39.322546 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424a78c2-8716-4127-a30c-1bac545564bb None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:39.323002 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424a78c2-8716-4127-a30c-1bac545564bb None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:39.323106 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424a78c2-8716-4127-a30c-1bac545564bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:39.371364 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:46:39.389286 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-424a78c2-8716-4127-a30c-1bac545564bb None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:46:39.390481 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-424a78c2-8716-4127-a30c-1bac545564bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:39.486096 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 717/1448] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 11 00:46:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:46:39.771455 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38b5db7-c321-45c0-ae55-e1f8c6d6ac48 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:46:39.773149 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-b38b5db7-c321-45c0-ae55-e1f8c6d6ac48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:39.874445 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38b5db7-c321-45c0-ae55-e1f8c6d6ac48 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:46:39.915156 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:46:39.950045 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15425132-1704-483b-9564-9266d9a51b81 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:46:39.950654 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15425132-1704-483b-9564-9266d9a51b81 service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:39.950967 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15425132-1704-483b-9564-9266d9a51b81 service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:39.951237 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15425132-1704-483b-9564-9266d9a51b81 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:39.952812 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-15425132-1704-483b-9564-9266d9a51b81 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:40.059386 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15425132-1704-483b-9564-9266d9a51b81 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:46:40.060741 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15425132-1704-483b-9564-9266d9a51b81 service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:46:40.079482 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 732/1449] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:46:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:46:42.369147 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09503e9a-6e90-4bd6-a183-052d665fb97b None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:42.369679 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09503e9a-6e90-4bd6-a183-052d665fb97b None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:42.369679 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09503e9a-6e90-4bd6-a183-052d665fb97b None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:42.370952 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 718/1450] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 11 00:46:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:46:42.376227 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b31f498-cb2d-4f0b-834c-ac6ede1728e2 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:42.376662 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b31f498-cb2d-4f0b-834c-ac6ede1728e2 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:42.376820 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b31f498-cb2d-4f0b-834c-ac6ede1728e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:42.442240 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-3b31f498-cb2d-4f0b-834c-ac6ede1728e2 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:46:42.444024 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-3b31f498-cb2d-4f0b-834c-ac6ede1728e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:42.464910 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:46:42.542697 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 733/1451] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 11 00:46:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:46:42.840766 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09503e9a-6e90-4bd6-a183-052d665fb97b None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:46:42.842276 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-09503e9a-6e90-4bd6-a183-052d665fb97b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:42.951120 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09503e9a-6e90-4bd6-a183-052d665fb97b None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:46:42.973719 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:46:43.028168 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee865c8-e939-470e-87c6-5ad6689d1339 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:46:43.028766 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee865c8-e939-470e-87c6-5ad6689d1339 service neutron] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:43.029044 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee865c8-e939-470e-87c6-5ad6689d1339 service neutron] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:43.029314 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee865c8-e939-470e-87c6-5ad6689d1339 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:43.030927 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-6ee865c8-e939-470e-87c6-5ad6689d1339 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:43.138548 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee865c8-e939-470e-87c6-5ad6689d1339 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:46:43.140029 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee865c8-e939-470e-87c6-5ad6689d1339 service neutron] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:46:43.158223 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 719/1452] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:46:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:46:45.394612 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10b7482-3d54-4a33-bd19-14bc29516d21 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:45.395183 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10b7482-3d54-4a33-bd19-14bc29516d21 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:45.395183 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10b7482-3d54-4a33-bd19-14bc29516d21 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:45.397323 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 734/1453] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 11 00:46:45 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:46:45.402648 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3300aa-6164-4a08-b30e-e65ae6bfc32b None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:45.403117 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3300aa-6164-4a08-b30e-e65ae6bfc32b None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:45.403215 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3300aa-6164-4a08-b30e-e65ae6bfc32b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:45.466075 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:46:45.469017 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-fc3300aa-6164-4a08-b30e-e65ae6bfc32b None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:46:45.470717 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-fc3300aa-6164-4a08-b30e-e65ae6bfc32b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:45.567851 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 720/1454] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 11 00:46:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:46:45.861223 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10b7482-3d54-4a33-bd19-14bc29516d21 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:46:45.862735 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-a10b7482-3d54-4a33-bd19-14bc29516d21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:45.965125 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10b7482-3d54-4a33-bd19-14bc29516d21 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:46:46.013307 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:46:46.039313 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-739066f8-fe7c-4f81-9349-75e034494114 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:46:46.039911 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739066f8-fe7c-4f81-9349-75e034494114 service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:46.040210 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739066f8-fe7c-4f81-9349-75e034494114 service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:46.040476 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739066f8-fe7c-4f81-9349-75e034494114 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:46.042036 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-739066f8-fe7c-4f81-9349-75e034494114 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:46.149896 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-739066f8-fe7c-4f81-9349-75e034494114 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:46:46.151317 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-739066f8-fe7c-4f81-9349-75e034494114 service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:46:46.169681 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 735/1455] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:46:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:46:48.419074 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca89838c-f9db-42a5-a4d0-ab78d53739aa None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:48.419608 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca89838c-f9db-42a5-a4d0-ab78d53739aa None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:48.419608 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca89838c-f9db-42a5-a4d0-ab78d53739aa None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:48.420901 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 721/1456] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 11 00:46:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:46:48.426040 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c06586-c003-40b6-a6af-2d69d613e957 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:48.426501 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c06586-c003-40b6-a6af-2d69d613e957 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:48.426599 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c06586-c003-40b6-a6af-2d69d613e957 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:48.492482 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-e1c06586-c003-40b6-a6af-2d69d613e957 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:46:48.493832 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-e1c06586-c003-40b6-a6af-2d69d613e957 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:48.524317 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:46:48.591296 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 736/1457] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 11 00:46:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:46:48.879336 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca89838c-f9db-42a5-a4d0-ab78d53739aa None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:46:48.881458 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-ca89838c-f9db-42a5-a4d0-ab78d53739aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:48.988677 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca89838c-f9db-42a5-a4d0-ab78d53739aa None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:46:49.018408 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:46:49.065048 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb9ae251-3246-4555-8d9b-94972d5ecc75 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:46:49.065649 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9ae251-3246-4555-8d9b-94972d5ecc75 service neutron] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:49.065923 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9ae251-3246-4555-8d9b-94972d5ecc75 service neutron] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:49.066188 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9ae251-3246-4555-8d9b-94972d5ecc75 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:49.067835 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-eb9ae251-3246-4555-8d9b-94972d5ecc75 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:49.177907 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb9ae251-3246-4555-8d9b-94972d5ecc75 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:46:49.179300 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb9ae251-3246-4555-8d9b-94972d5ecc75 service neutron] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:46:49.197280 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 722/1458] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:46:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:46:51.422608 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9ea5a0-9469-4841-a494-fa638eae54be None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:51.423180 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9ea5a0-9469-4841-a494-fa638eae54be None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:51.423180 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9ea5a0-9469-4841-a494-fa638eae54be None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:51.424475 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 737/1459] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 11 00:46:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:46:51.430301 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b53b6c-f316-4833-b5e6-3249d7bb50ce None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:51.430722 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b53b6c-f316-4833-b5e6-3249d7bb50ce None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:51.430887 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b53b6c-f316-4833-b5e6-3249d7bb50ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:51.497129 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-77b53b6c-f316-4833-b5e6-3249d7bb50ce None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:46:51.498375 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-77b53b6c-f316-4833-b5e6-3249d7bb50ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:51.509112 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:46:51.599386 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 723/1460] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 11 00:46:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:46:51.886875 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e9ea5a0-9469-4841-a494-fa638eae54be None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:46:51.888389 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-1e9ea5a0-9469-4841-a494-fa638eae54be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:51.992586 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e9ea5a0-9469-4841-a494-fa638eae54be None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:46:52.046780 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:46:52.064886 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c123b1-245a-42b7-87b2-fe87c7fea5db service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:46:52.065487 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c123b1-245a-42b7-87b2-fe87c7fea5db service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:52.065761 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c123b1-245a-42b7-87b2-fe87c7fea5db service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:52.066027 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c123b1-245a-42b7-87b2-fe87c7fea5db service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:52.068182 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-18c123b1-245a-42b7-87b2-fe87c7fea5db service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:52.177238 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18c123b1-245a-42b7-87b2-fe87c7fea5db service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:46:52.178698 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18c123b1-245a-42b7-87b2-fe87c7fea5db service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:46:52.196366 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 738/1461] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:46:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:46:54.427468 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774854ef-2e65-4fd4-915e-86b6f41d59cf None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:54.428114 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774854ef-2e65-4fd4-915e-86b6f41d59cf None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:54.428114 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774854ef-2e65-4fd4-915e-86b6f41d59cf None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:54.429346 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 724/1462] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 11 00:46:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:46:54.435308 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7953c462-d6e9-4d83-9795-476c956f23a7 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:54.435763 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7953c462-d6e9-4d83-9795-476c956f23a7 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:54.435870 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7953c462-d6e9-4d83-9795-476c956f23a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:54.501578 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-7953c462-d6e9-4d83-9795-476c956f23a7 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:46:54.502771 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-7953c462-d6e9-4d83-9795-476c956f23a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:54.540611 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:46:54.598028 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 739/1463] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 11 00:46:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:46:54.890605 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-774854ef-2e65-4fd4-915e-86b6f41d59cf None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:46:54.892173 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-774854ef-2e65-4fd4-915e-86b6f41d59cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:54.996474 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-774854ef-2e65-4fd4-915e-86b6f41d59cf None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:46:55.035269 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:46:55.073036 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c35e83-1ffb-4490-b09e-73492f6f7125 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:46:55.073634 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c35e83-1ffb-4490-b09e-73492f6f7125 service neutron] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:55.073905 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c35e83-1ffb-4490-b09e-73492f6f7125 service neutron] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:55.074169 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c35e83-1ffb-4490-b09e-73492f6f7125 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:55.075769 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-14c35e83-1ffb-4490-b09e-73492f6f7125 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:55.182983 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14c35e83-1ffb-4490-b09e-73492f6f7125 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:46:55.184352 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14c35e83-1ffb-4490-b09e-73492f6f7125 service neutron] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:46:55.202923 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 725/1464] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:46:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:46:57.366511 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fcc10a4-448f-472b-988f-574072addcc7 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:57.367213 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fcc10a4-448f-472b-988f-574072addcc7 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:57.367213 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fcc10a4-448f-472b-988f-574072addcc7 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:57.368461 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 740/1465] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 11 00:46:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:46:57.373841 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-853cc374-374b-4510-8550-e350b6c74249 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:57.374271 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-853cc374-374b-4510-8550-e350b6c74249 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:57.374379 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-853cc374-374b-4510-8550-e350b6c74249 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:57.440882 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-853cc374-374b-4510-8550-e350b6c74249 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:46:57.442098 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-853cc374-374b-4510-8550-e350b6c74249 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:57.461162 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:46:57.540252 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 726/1466] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 11 00:46:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:46:57.826281 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fcc10a4-448f-472b-988f-574072addcc7 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:46:57.827854 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-9fcc10a4-448f-472b-988f-574072addcc7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:57.935618 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fcc10a4-448f-472b-988f-574072addcc7 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:46:58.002660 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:46:58.010118 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de396668-7613-4626-9e05-be6d43f67bb1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:46:58.010727 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de396668-7613-4626-9e05-be6d43f67bb1 service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:46:58.011032 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de396668-7613-4626-9e05-be6d43f67bb1 service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:46:58.011307 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de396668-7613-4626-9e05-be6d43f67bb1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:46:58.012870 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-de396668-7613-4626-9e05-be6d43f67bb1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:46:58.120430 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de396668-7613-4626-9e05-be6d43f67bb1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:46:58.121855 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de396668-7613-4626-9e05-be6d43f67bb1 service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:46:58.140043 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 741/1467] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:46:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:47:00.307758 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4da31dd-c2df-4af6-ba6b-2b0b8cb77e34 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:47:00.308351 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4da31dd-c2df-4af6-ba6b-2b0b8cb77e34 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:47:00.308351 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4da31dd-c2df-4af6-ba6b-2b0b8cb77e34 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:47:00.309636 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 727/1468] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 11 00:47:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:47:00.315027 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7341134-b633-488a-b893-ab80efea4072 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:47:00.315453 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7341134-b633-488a-b893-ab80efea4072 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:47:00.315551 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7341134-b633-488a-b893-ab80efea4072 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:47:00.381036 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-c7341134-b633-488a-b893-ab80efea4072 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:47:00.382146 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-c7341134-b633-488a-b893-ab80efea4072 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:47:00.429553 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:47:00.476229 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 742/1469] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 11 00:47:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:47:00.757580 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4da31dd-c2df-4af6-ba6b-2b0b8cb77e34 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:47:00.759130 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-e4da31dd-c2df-4af6-ba6b-2b0b8cb77e34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:47:00.862493 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4da31dd-c2df-4af6-ba6b-2b0b8cb77e34 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:47:00.910012 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:47:00.937529 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17869dd8-86a4-4ddf-aaf2-9994dfeffff6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:47:00.938134 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17869dd8-86a4-4ddf-aaf2-9994dfeffff6 service neutron] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:47:00.938408 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17869dd8-86a4-4ddf-aaf2-9994dfeffff6 service neutron] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:47:00.938673 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17869dd8-86a4-4ddf-aaf2-9994dfeffff6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:47:00.940770 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-17869dd8-86a4-4ddf-aaf2-9994dfeffff6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:47:01.049760 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17869dd8-86a4-4ddf-aaf2-9994dfeffff6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:47:01.051145 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17869dd8-86a4-4ddf-aaf2-9994dfeffff6 service neutron] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:47:01.069022 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 728/1470] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:47:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:47:10.474690 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=22 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:47:11.066448 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=66 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:47:20.981282 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25be0189-96cc-4818-ba09-179abaf7b3dd None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:47:20.981821 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25be0189-96cc-4818-ba09-179abaf7b3dd None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:47:20.981821 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25be0189-96cc-4818-ba09-179abaf7b3dd None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:47:20.983104 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 743/1471] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:47:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:47:20.988410 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23a95ff-edb6-4a34-baa3-17f3fc2954e2 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:47:20.989450 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23a95ff-edb6-4a34-baa3-17f3fc2954e2 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:47:20.989566 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23a95ff-edb6-4a34-baa3-17f3fc2954e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:47:21.057122 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-a23a95ff-edb6-4a34-baa3-17f3fc2954e2 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:47:21.058456 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-a23a95ff-edb6-4a34-baa3-17f3fc2954e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:47:21.154644 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 729/1472] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:47:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:47:21.459739 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9021764-2559-43a7-954d-25a9eb43f0d7 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:47:21.460023 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9021764-2559-43a7-954d-25a9eb43f0d7 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:47:21.460294 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9021764-2559-43a7-954d-25a9eb43f0d7 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:47:21.461621 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 744/1473] 10.222.0.27 () {54 vars in 966 bytes} [Sun Sep 11 00:47:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:47:21.466506 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a402c7e-6c6b-4efd-b864-6e55c6ad6a20 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:47:21.466771 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a402c7e-6c6b-4efd-b864-6e55c6ad6a20 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:47:21.467077 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a402c7e-6c6b-4efd-b864-6e55c6ad6a20 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:47:21.544487 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-0a402c7e-6c6b-4efd-b864-6e55c6ad6a20 None None] MFA Rules not processed for user `06b1c911731f4267a5f37e91d948e7e9`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:47:21.545609 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-0a402c7e-6c6b-4efd-b864-6e55c6ad6a20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:47:21.566054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:47:21.624144 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 730/1474] 10.222.0.27 () {58 vars in 1063 bytes} [Sun Sep 11 00:47:21 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:47:21.631803 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9021764-2559-43a7-954d-25a9eb43f0d7 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:47:21.633358 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-c9021764-2559-43a7-954d-25a9eb43f0d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:47:21.736599 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9021764-2559-43a7-954d-25a9eb43f0d7 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:47:21.811077 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-968a8b82-a692-4cee-b9e2-ae2fbcc32210 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '06b1c911731f4267a5f37e91d948e7e9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:47:21.811606 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968a8b82-a692-4cee-b9e2-ae2fbcc32210 service glance] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:47:21.811885 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968a8b82-a692-4cee-b9e2-ae2fbcc32210 service glance] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:47:21.812156 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968a8b82-a692-4cee-b9e2-ae2fbcc32210 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:47:21.813727 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-968a8b82-a692-4cee-b9e2-ae2fbcc32210 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:47:21.867248 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:47:21.921584 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-968a8b82-a692-4cee-b9e2-ae2fbcc32210 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:47:21.923043 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-968a8b82-a692-4cee-b9e2-ae2fbcc32210 service glance] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:47:21.941127 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 745/1475] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:47:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 11 00:47:24.175770 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90ddd20-8831-4f50-b93e-2ec18c7a4010 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:47:24.176547 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90ddd20-8831-4f50-b93e-2ec18c7a4010 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:47:24.176547 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90ddd20-8831-4f50-b93e-2ec18c7a4010 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:47:24.177772 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 731/1476] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:47:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:47:24.183144 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec8bfae-07d6-4acf-b71f-0d26b1b06928 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:47:24.183604 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec8bfae-07d6-4acf-b71f-0d26b1b06928 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:47:24.183710 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec8bfae-07d6-4acf-b71f-0d26b1b06928 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:47:24.249269 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-fec8bfae-07d6-4acf-b71f-0d26b1b06928 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:47:24.250460 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-fec8bfae-07d6-4acf-b71f-0d26b1b06928 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:47:24.345877 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 746/1477] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:47:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:47:24.619262 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e90ddd20-8831-4f50-b93e-2ec18c7a4010 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:47:24.620789 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-e90ddd20-8831-4f50-b93e-2ec18c7a4010 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:47:24.749582 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e90ddd20-8831-4f50-b93e-2ec18c7a4010 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:47:24.800470 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:47:24.845366 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11b91e34-a6c2-4f5e-b8c5-bd735b936648 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:47:24.845964 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b91e34-a6c2-4f5e-b8c5-bd735b936648 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:47:24.846236 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b91e34-a6c2-4f5e-b8c5-bd735b936648 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:47:24.846501 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b91e34-a6c2-4f5e-b8c5-bd735b936648 admin admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:47:24.847167 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11b91e34-a6c2-4f5e-b8c5-bd735b936648 admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:47:24.849569 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: WARNING py.warnings [None req-11b91e34-a6c2-4f5e-b8c5-bd735b936648 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 11 00:47:24.849569 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: warnings.warn(msg) Sep 11 00:47:24.849569 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: Sep 11 00:47:24.850706 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11b91e34-a6c2-4f5e-b8c5-bd735b936648 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:47:24.869691 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 732/1478] 10.222.0.27 () {56 vars in 1108 bytes} [Sun Sep 11 00:47:24 2022] GET /identity/v3/projects => generated 2553 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:47:27.049507 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1324748-a1ce-465f-ad12-f2e78239f34b None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:47:27.050109 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1324748-a1ce-465f-ad12-f2e78239f34b None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:47:27.050109 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1324748-a1ce-465f-ad12-f2e78239f34b None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:47:27.052082 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 747/1479] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:47:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:47:27.057632 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d21c69-12ed-4e40-b5ba-ba3d8eef0874 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:47:27.058099 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d21c69-12ed-4e40-b5ba-ba3d8eef0874 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:47:27.058229 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d21c69-12ed-4e40-b5ba-ba3d8eef0874 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:47:27.126003 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-81d21c69-12ed-4e40-b5ba-ba3d8eef0874 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:47:27.127277 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-81d21c69-12ed-4e40-b5ba-ba3d8eef0874 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:47:27.224761 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 733/1480] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:47:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:47:27.518820 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1324748-a1ce-465f-ad12-f2e78239f34b None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:47:27.521165 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-d1324748-a1ce-465f-ad12-f2e78239f34b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:47:27.530999 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:47:27.666925 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1324748-a1ce-465f-ad12-f2e78239f34b None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:47:27.745291 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:47:27.780116 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb0406d-7f93-4675-8dfe-b2c3ce189080 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '835def0e6c6d4365a2767051509eb0d6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'ResellerAdmin', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:47:27.780671 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb0406d-7f93-4675-8dfe-b2c3ce189080 service nova] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:47:27.780926 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb0406d-7f93-4675-8dfe-b2c3ce189080 service nova] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:47:27.781404 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb0406d-7f93-4675-8dfe-b2c3ce189080 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:47:27.783179 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-1bb0406d-7f93-4675-8dfe-b2c3ce189080 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:47:27.871574 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bb0406d-7f93-4675-8dfe-b2c3ce189080 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:47:27.873200 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bb0406d-7f93-4675-8dfe-b2c3ce189080 service nova] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:47:27.891811 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 748/1481] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:47:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:47:30.262085 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b9aba3-68a0-41b8-8f70-69c3d2922514 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:47:30.262621 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b9aba3-68a0-41b8-8f70-69c3d2922514 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:47:30.262621 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b9aba3-68a0-41b8-8f70-69c3d2922514 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:47:30.264007 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 734/1482] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:47:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:47:30.269386 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b4a6ea-b5b8-4e88-983c-499d805d2fc7 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:47:30.270358 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b4a6ea-b5b8-4e88-983c-499d805d2fc7 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:47:30.270711 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b4a6ea-b5b8-4e88-983c-499d805d2fc7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:47:30.336625 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-03b4a6ea-b5b8-4e88-983c-499d805d2fc7 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:47:30.337831 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-03b4a6ea-b5b8-4e88-983c-499d805d2fc7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:47:30.383204 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:47:30.433689 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 749/1483] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:47:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:47:30.728050 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b9aba3-68a0-41b8-8f70-69c3d2922514 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:47:30.729599 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-87b9aba3-68a0-41b8-8f70-69c3d2922514 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:47:30.753785 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:47:30.833895 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b9aba3-68a0-41b8-8f70-69c3d2922514 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:47:30.907038 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6712d374-079b-495e-96b1-527b5b27f97a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:47:30.907638 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6712d374-079b-495e-96b1-527b5b27f97a service neutron] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:47:30.907917 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6712d374-079b-495e-96b1-527b5b27f97a service neutron] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:47:30.908189 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6712d374-079b-495e-96b1-527b5b27f97a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:47:30.910032 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-6712d374-079b-495e-96b1-527b5b27f97a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:47:31.101145 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:47:31.142469 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6712d374-079b-495e-96b1-527b5b27f97a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:47:31.143950 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6712d374-079b-495e-96b1-527b5b27f97a service neutron] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:47:31.162972 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 735/1484] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:47:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 435 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:47:33.384288 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df74473-e233-43e8-a324-57d52a39921e None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:47:33.384806 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df74473-e233-43e8-a324-57d52a39921e None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:47:33.384806 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df74473-e233-43e8-a324-57d52a39921e None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:47:33.386089 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 750/1485] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:47:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:47:33.391316 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9759b6a6-c577-4fe4-945b-67bc37a5c21e None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:47:33.391740 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9759b6a6-c577-4fe4-945b-67bc37a5c21e None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:47:33.391864 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9759b6a6-c577-4fe4-945b-67bc37a5c21e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:47:33.457566 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-9759b6a6-c577-4fe4-945b-67bc37a5c21e None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:47:33.458759 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-9759b6a6-c577-4fe4-945b-67bc37a5c21e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:47:33.555648 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 736/1486] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:47:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:47:33.846192 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df74473-e233-43e8-a324-57d52a39921e None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:47:33.847774 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-2df74473-e233-43e8-a324-57d52a39921e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:47:33.953323 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df74473-e233-43e8-a324-57d52a39921e None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:47:34.025014 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:47:34.026664 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e500e2-6ef3-4410-afb7-7b9033bc2143 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:47:34.027376 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e500e2-6ef3-4410-afb7-7b9033bc2143 service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:47:34.027914 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e500e2-6ef3-4410-afb7-7b9033bc2143 service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:47:34.028385 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e500e2-6ef3-4410-afb7-7b9033bc2143 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:47:34.030153 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-e5e500e2-6ef3-4410-afb7-7b9033bc2143 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:47:34.139349 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5e500e2-6ef3-4410-afb7-7b9033bc2143 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:47:34.140779 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5e500e2-6ef3-4410-afb7-7b9033bc2143 service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:47:34.158803 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 751/1487] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:47:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:47:36.595903 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bddd6e4f-5c7b-4dca-882e-de7951ec67cf None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:47:36.596485 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bddd6e4f-5c7b-4dca-882e-de7951ec67cf None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:47:36.596485 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bddd6e4f-5c7b-4dca-882e-de7951ec67cf None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:47:36.597846 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 737/1488] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:47:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:47:36.603520 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21467365-3ef9-4d5d-97c0-63f5bab992c6 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:47:36.604033 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21467365-3ef9-4d5d-97c0-63f5bab992c6 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:47:36.604144 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21467365-3ef9-4d5d-97c0-63f5bab992c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:47:36.669086 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-21467365-3ef9-4d5d-97c0-63f5bab992c6 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:47:36.670280 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-21467365-3ef9-4d5d-97c0-63f5bab992c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:47:36.722116 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:47:36.765497 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 752/1489] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:47:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:47:37.057936 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bddd6e4f-5c7b-4dca-882e-de7951ec67cf None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:47:37.059480 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-bddd6e4f-5c7b-4dca-882e-de7951ec67cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:47:37.097660 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:47:37.170445 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bddd6e4f-5c7b-4dca-882e-de7951ec67cf None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:47:37.243851 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d94809-bcdc-440b-afe1-30f948972204 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:47:37.244447 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d94809-bcdc-440b-afe1-30f948972204 service neutron] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:47:37.244722 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d94809-bcdc-440b-afe1-30f948972204 service neutron] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:47:37.244988 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d94809-bcdc-440b-afe1-30f948972204 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:47:37.246565 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-58d94809-bcdc-440b-afe1-30f948972204 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:47:37.325764 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:47:37.355248 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58d94809-bcdc-440b-afe1-30f948972204 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:47:37.356698 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58d94809-bcdc-440b-afe1-30f948972204 service neutron] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:47:37.374670 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 738/1490] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:47:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 11 00:47:39.790959 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a4c867-09ac-4e81-98a5-756fcec30d02 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:47:39.791624 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a4c867-09ac-4e81-98a5-756fcec30d02 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:47:39.791624 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a4c867-09ac-4e81-98a5-756fcec30d02 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:47:39.792854 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 753/1491] 10.222.0.27 () {54 vars in 927 bytes} [Sun Sep 11 00:47:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:47:39.798117 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75fe8a28-cd75-4efc-af17-dcba8836b6b2 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:47:39.798578 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75fe8a28-cd75-4efc-af17-dcba8836b6b2 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:47:39.798687 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75fe8a28-cd75-4efc-af17-dcba8836b6b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:47:39.864348 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-75fe8a28-cd75-4efc-af17-dcba8836b6b2 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:47:39.865453 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-75fe8a28-cd75-4efc-af17-dcba8836b6b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:47:39.961891 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 739/1492] 10.222.0.27 () {58 vars in 1024 bytes} [Sun Sep 11 00:47:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:47:40.253377 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a4c867-09ac-4e81-98a5-756fcec30d02 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:47:40.256759 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-34a4c867-09ac-4e81-98a5-756fcec30d02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:47:40.379930 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a4c867-09ac-4e81-98a5-756fcec30d02 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:47:40.453649 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7684403b-ec1f-4419-adeb-7eb9bb00a93a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:47:40.454256 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7684403b-ec1f-4419-adeb-7eb9bb00a93a service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:47:40.454540 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7684403b-ec1f-4419-adeb-7eb9bb00a93a service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:47:40.454832 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7684403b-ec1f-4419-adeb-7eb9bb00a93a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:47:40.456454 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-7684403b-ec1f-4419-adeb-7eb9bb00a93a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:47:40.465882 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:47:40.575709 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7684403b-ec1f-4419-adeb-7eb9bb00a93a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:47:40.577671 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7684403b-ec1f-4419-adeb-7eb9bb00a93a service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:47:40.603182 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 754/1493] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:47:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 350 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:47:49.960321 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=82 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:47:50.600040 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=50 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:49:00.825069 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8760e43f-8d41-44e5-90a8-e86a7b0fbb1c None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:49:00.825682 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8760e43f-8d41-44e5-90a8-e86a7b0fbb1c None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:49:00.825682 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8760e43f-8d41-44e5-90a8-e86a7b0fbb1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:49:00.901265 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-8760e43f-8d41-44e5-90a8-e86a7b0fbb1c None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:49:00.902459 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-8760e43f-8d41-44e5-90a8-e86a7b0fbb1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:49:01.000969 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 740/1494] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:49:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:49:01.009733 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42263825-919f-4607-8280-fe87e2cd8a21 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:49:01.010239 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42263825-919f-4607-8280-fe87e2cd8a21 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:49:01.010346 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42263825-919f-4607-8280-fe87e2cd8a21 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:49:01.085944 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-42263825-919f-4607-8280-fe87e2cd8a21 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:49:01.087108 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-42263825-919f-4607-8280-fe87e2cd8a21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:49:01.183241 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 755/1495] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:49:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:49:01.187933 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8760e43f-8d41-44e5-90a8-e86a7b0fbb1c None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:49:01.189128 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-8760e43f-8d41-44e5-90a8-e86a7b0fbb1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:49:01.200431 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8760e43f-8d41-44e5-90a8-e86a7b0fbb1c None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:49:01.260625 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:49:01.294425 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8c2532-14f4-42d9-b5b8-705d043c30b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:49:01.295059 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8c2532-14f4-42d9-b5b8-705d043c30b8 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:49:01.295336 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8c2532-14f4-42d9-b5b8-705d043c30b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:49:01.295603 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8c2532-14f4-42d9-b5b8-705d043c30b8 admin admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:49:01.296244 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be8c2532-14f4-42d9-b5b8-705d043c30b8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:49:01.296545 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be8c2532-14f4-42d9-b5b8-705d043c30b8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:49:01.297568 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: WARNING py.warnings [None req-be8c2532-14f4-42d9-b5b8-705d043c30b8 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 11 00:49:01.297568 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: warnings.warn(msg) Sep 11 00:49:01.297568 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: Sep 11 00:49:01.298448 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be8c2532-14f4-42d9-b5b8-705d043c30b8 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:49:01.310191 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 741/1496] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 00:49:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 123 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:49:01.318498 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42263825-919f-4607-8280-fe87e2cd8a21 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:49:01.327709 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42263825-919f-4607-8280-fe87e2cd8a21 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:49:01.388317 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:49:01.422299 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf3d908-d485-44a2-a7dc-fb90653bb03b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:49:01.422931 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf3d908-d485-44a2-a7dc-fb90653bb03b admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:49:01.423211 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf3d908-d485-44a2-a7dc-fb90653bb03b admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:49:01.423484 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf3d908-d485-44a2-a7dc-fb90653bb03b admin admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:49:01.424122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faf3d908-d485-44a2-a7dc-fb90653bb03b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:49:01.426565 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: WARNING py.warnings [None req-faf3d908-d485-44a2-a7dc-fb90653bb03b admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 11 00:49:01.426565 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: warnings.warn(msg) Sep 11 00:49:01.426565 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: Sep 11 00:49:01.427781 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faf3d908-d485-44a2-a7dc-fb90653bb03b admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:49:01.483926 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 756/1497] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:49:01 2022] POST /identity/v3/projects => generated 379 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:49:01.488620 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8c2532-14f4-42d9-b5b8-705d043c30b8 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:49:01.490110 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-be8c2532-14f4-42d9-b5b8-705d043c30b8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:49:01.525471 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8c2532-14f4-42d9-b5b8-705d043c30b8 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:49:01.621357 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdd6935f-b598-4358-ad0f-6ab52eac64a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:49:01.621964 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd6935f-b598-4358-ad0f-6ab52eac64a7 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:49:01.622241 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd6935f-b598-4358-ad0f-6ab52eac64a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:49:01.622507 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd6935f-b598-4358-ad0f-6ab52eac64a7 admin admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:49:01.623170 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdd6935f-b598-4358-ad0f-6ab52eac64a7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:49:01.624191 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: WARNING py.warnings [None req-cdd6935f-b598-4358-ad0f-6ab52eac64a7 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 11 00:49:01.624191 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: warnings.warn(msg) Sep 11 00:49:01.624191 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: Sep 11 00:49:01.625351 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdd6935f-b598-4358-ad0f-6ab52eac64a7 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:49:01.667920 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 742/1498] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:49:01 2022] POST /identity/v3/users => generated 290 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:49:01.672002 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf3d908-d485-44a2-a7dc-fb90653bb03b admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:49:01.673591 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-faf3d908-d485-44a2-a7dc-fb90653bb03b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:49:01.735368 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf3d908-d485-44a2-a7dc-fb90653bb03b admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:49:01.818832 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:49:01.829814 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c1d545e-4a21-4081-bb3f-087e6d97f738 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:49:01.830412 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1d545e-4a21-4081-bb3f-087e6d97f738 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:49:01.830687 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1d545e-4a21-4081-bb3f-087e6d97f738 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:49:01.830984 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1d545e-4a21-4081-bb3f-087e6d97f738 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:49:01.831649 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c1d545e-4a21-4081-bb3f-087e6d97f738 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:49:01.832656 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: WARNING py.warnings [None req-1c1d545e-4a21-4081-bb3f-087e6d97f738 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 11 00:49:01.832656 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: warnings.warn(msg) Sep 11 00:49:01.832656 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: Sep 11 00:49:01.833582 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c1d545e-4a21-4081-bb3f-087e6d97f738 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:49:01.845710 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 757/1499] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:49:01 2022] GET /identity/v3/roles => generated 1420 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:49:01.850071 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdd6935f-b598-4358-ad0f-6ab52eac64a7 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:49:01.851714 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-cdd6935f-b598-4358-ad0f-6ab52eac64a7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:49:01.863464 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdd6935f-b598-4358-ad0f-6ab52eac64a7 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:49:01.888558 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:49:01.961575 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd17467-aea1-4463-948a-003fdfe78c2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:49:01.962174 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd17467-aea1-4463-948a-003fdfe78c2c admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:49:01.962454 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd17467-aea1-4463-948a-003fdfe78c2c admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:49:01.962721 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd17467-aea1-4463-948a-003fdfe78c2c admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:49:01.963415 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fd17467-aea1-4463-948a-003fdfe78c2c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:49:01.964426 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: WARNING py.warnings [None req-7fd17467-aea1-4463-948a-003fdfe78c2c admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 11 00:49:01.964426 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: warnings.warn(msg) Sep 11 00:49:01.964426 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: Sep 11 00:49:01.965300 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fd17467-aea1-4463-948a-003fdfe78c2c admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:49:01.978150 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 743/1500] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:49:01 2022] GET /identity/v3/roles => generated 1420 bytes in 129 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:49:01.982372 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c1d545e-4a21-4081-bb3f-087e6d97f738 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:49:01.983907 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-1c1d545e-4a21-4081-bb3f-087e6d97f738 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:49:02.017880 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c1d545e-4a21-4081-bb3f-087e6d97f738 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:49:02.110238 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40571739-5a9f-4ae4-93a3-369919d53e13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:49:02.110936 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40571739-5a9f-4ae4-93a3-369919d53e13 admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:49:02.111215 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40571739-5a9f-4ae4-93a3-369919d53e13 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:49:02.111484 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40571739-5a9f-4ae4-93a3-369919d53e13 admin admin] PATH_INFO: `/v3/projects/6bd23ad8458f40828b8071193b91a99d/users/a502e1eb3bef4ba7ad1482a3855d8e83/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:49:02.133864 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40571739-5a9f-4ae4-93a3-369919d53e13 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6bd23ad8458f40828b8071193b91a99d, user_id=a502e1eb3bef4ba7ad1482a3855d8e83, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:49:02.134919 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: WARNING py.warnings [None req-40571739-5a9f-4ae4-93a3-369919d53e13 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 11 00:49:02.134919 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: warnings.warn(msg) Sep 11 00:49:02.134919 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: Sep 11 00:49:02.136851 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40571739-5a9f-4ae4-93a3-369919d53e13 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:49:02.161636 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 758/1501] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:49:01 2022] PUT /identity/v3/projects/6bd23ad8458f40828b8071193b91a99d/users/a502e1eb3bef4ba7ad1482a3855d8e83/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:49:02.173830 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69f42f9-7890-4c1c-bc6e-b00db8080d9f None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:49:02.174117 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69f42f9-7890-4c1c-bc6e-b00db8080d9f None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:49:02.174393 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69f42f9-7890-4c1c-bc6e-b00db8080d9f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:49:02.231420 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-b69f42f9-7890-4c1c-bc6e-b00db8080d9f None None] MFA Rules not processed for user `a502e1eb3bef4ba7ad1482a3855d8e83`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:49:02.232537 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-b69f42f9-7890-4c1c-bc6e-b00db8080d9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:49:02.314805 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 744/1502] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:49:02 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:49:02.329657 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a8de94-89bd-4634-a412-799c89308b59 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:49:02.330109 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a8de94-89bd-4634-a412-799c89308b59 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:49:02.330465 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a8de94-89bd-4634-a412-799c89308b59 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:49:02.332898 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 759/1503] 10.222.0.27 () {54 vars in 964 bytes} [Sun Sep 11 00:49:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 11 00:49:02.337970 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217bd03c-98d1-47de-b9d8-0451ee32ddaa None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:49:02.338252 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217bd03c-98d1-47de-b9d8-0451ee32ddaa None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:49:02.338520 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217bd03c-98d1-47de-b9d8-0451ee32ddaa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:49:02.370566 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:49:02.414856 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-217bd03c-98d1-47de-b9d8-0451ee32ddaa None None] MFA Rules not processed for user `22374eaff0834245b88506dcd7b24ede`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:49:02.416047 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-217bd03c-98d1-47de-b9d8-0451ee32ddaa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:49:02.490347 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 745/1504] 10.222.0.27 () {58 vars in 1061 bytes} [Sun Sep 11 00:49:02 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:49:02.497933 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a8de94-89bd-4634-a412-799c89308b59 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:49:02.499459 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-93a8de94-89bd-4634-a412-799c89308b59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:49:02.550200 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:49:02.603407 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a8de94-89bd-4634-a412-799c89308b59 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:49:02.676439 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60d5250-9e20-4c37-ac72-22048ea3698d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:49:02.677040 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60d5250-9e20-4c37-ac72-22048ea3698d service cinder] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:49:02.677319 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60d5250-9e20-4c37-ac72-22048ea3698d service cinder] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:49:02.677592 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60d5250-9e20-4c37-ac72-22048ea3698d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:49:02.679212 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-a60d5250-9e20-4c37-ac72-22048ea3698d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:49:02.777314 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a60d5250-9e20-4c37-ac72-22048ea3698d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:49:02.778661 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a60d5250-9e20-4c37-ac72-22048ea3698d service cinder] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:49:02.782069 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:49:02.797630 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 760/1505] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:49:02 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 300 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:49:02.844164 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d994ff2-f4e0-4e06-83a5-a604bb3c9bde None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:49:02.844448 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d994ff2-f4e0-4e06-83a5-a604bb3c9bde None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:49:02.844818 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d994ff2-f4e0-4e06-83a5-a604bb3c9bde None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:49:02.846108 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 746/1506] 10.222.0.27 () {52 vars in 830 bytes} [Sun Sep 11 00:49:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:49:02.849759 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60d5250-9e20-4c37-ac72-22048ea3698d service cinder] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:49:02.851012 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-a60d5250-9e20-4c37-ac72-22048ea3698d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:49:02.949359 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60d5250-9e20-4c37-ac72-22048ea3698d service cinder] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:49:03.040684 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df2f0edf-bd17-4246-9944-8a5db46be6aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:49:03.041298 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2f0edf-bd17-4246-9944-8a5db46be6aa admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:49:03.041485 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2f0edf-bd17-4246-9944-8a5db46be6aa admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:49:03.041747 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2f0edf-bd17-4246-9944-8a5db46be6aa admin admin] PATH_INFO: `/v3/users/a502e1eb3bef4ba7ad1482a3855d8e83` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:49:03.054800 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df2f0edf-bd17-4246-9944-8a5db46be6aa admin admin] RBAC: Authorizing `identity:delete_user(user_id=a502e1eb3bef4ba7ad1482a3855d8e83)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:49:03.055828 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: WARNING py.warnings [None req-df2f0edf-bd17-4246-9944-8a5db46be6aa admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 11 00:49:03.055828 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: warnings.warn(msg) Sep 11 00:49:03.055828 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: Sep 11 00:49:03.057019 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df2f0edf-bd17-4246-9944-8a5db46be6aa admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:49:03.067078 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:49:03.126760 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-df2f0edf-bd17-4246-9944-8a5db46be6aa admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a502e1eb3bef4ba7ad1482a3855d8e83'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:49:03.127223 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-df2f0edf-bd17-4246-9944-8a5db46be6aa admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a502e1eb3bef4ba7ad1482a3855d8e83'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:49:03.127738 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-df2f0edf-bd17-4246-9944-8a5db46be6aa admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a502e1eb3bef4ba7ad1482a3855d8e83'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:49:03.141712 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-df2f0edf-bd17-4246-9944-8a5db46be6aa admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a502e1eb3bef4ba7ad1482a3855d8e83'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:49:03.164360 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-df2f0edf-bd17-4246-9944-8a5db46be6aa admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a502e1eb3bef4ba7ad1482a3855d8e83'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:49:03.179384 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 761/1507] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:49:02 2022] DELETE /identity/v3/users/a502e1eb3bef4ba7ad1482a3855d8e83 => generated 0 bytes in 330 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:49:03.183792 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a41184-9520-48ce-9abc-68a8852618db None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:49:03.184286 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a41184-9520-48ce-9abc-68a8852618db None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:49:03.184395 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a41184-9520-48ce-9abc-68a8852618db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:49:03.241689 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-e1a41184-9520-48ce-9abc-68a8852618db None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:49:03.242895 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-e1a41184-9520-48ce-9abc-68a8852618db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:49:03.319235 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:49:03.332549 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 747/1508] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:49:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:49:03.342877 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df2f0edf-bd17-4246-9944-8a5db46be6aa admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:49:03.344099 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-df2f0edf-bd17-4246-9944-8a5db46be6aa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:49:03.445843 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df2f0edf-bd17-4246-9944-8a5db46be6aa admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:49:03.500433 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:49:03.520527 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da1ec155-9e32-48b5-b96c-48480d9850f9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:49:03.521138 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1ec155-9e32-48b5-b96c-48480d9850f9 service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:49:03.521411 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1ec155-9e32-48b5-b96c-48480d9850f9 service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:49:03.521679 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1ec155-9e32-48b5-b96c-48480d9850f9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:49:03.523314 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-da1ec155-9e32-48b5-b96c-48480d9850f9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:49:03.639115 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da1ec155-9e32-48b5-b96c-48480d9850f9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:49:03.640562 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da1ec155-9e32-48b5-b96c-48480d9850f9 service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:49:03.658212 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 762/1509] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:49:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:49:03.984091 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1a41184-9520-48ce-9abc-68a8852618db None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:49:03.985325 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-e1a41184-9520-48ce-9abc-68a8852618db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:49:03.997410 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1a41184-9520-48ce-9abc-68a8852618db None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:49:04.098610 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc38e17-5eaf-41dd-8210-35fa63aeee06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:49:04.099263 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc38e17-5eaf-41dd-8210-35fa63aeee06 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:49:04.099543 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc38e17-5eaf-41dd-8210-35fa63aeee06 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:49:04.099809 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc38e17-5eaf-41dd-8210-35fa63aeee06 admin admin] PATH_INFO: `/v3/projects/6bd23ad8458f40828b8071193b91a99d` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:49:04.109908 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dc38e17-5eaf-41dd-8210-35fa63aeee06 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6bd23ad8458f40828b8071193b91a99d)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:49:04.110972 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: WARNING py.warnings [None req-8dc38e17-5eaf-41dd-8210-35fa63aeee06 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 11 00:49:04.110972 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: warnings.warn(msg) Sep 11 00:49:04.110972 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: Sep 11 00:49:04.112161 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dc38e17-5eaf-41dd-8210-35fa63aeee06 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:49:04.171127 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-8dc38e17-5eaf-41dd-8210-35fa63aeee06 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6bd23ad8458f40828b8071193b91a99d', 'request_id': 'req-8dc38e17-5eaf-41dd-8210-35fa63aeee06'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:49:04.184816 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-8dc38e17-5eaf-41dd-8210-35fa63aeee06 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6bd23ad8458f40828b8071193b91a99d', 'request_id': 'req-8dc38e17-5eaf-41dd-8210-35fa63aeee06'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:49:04.190945 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-8dc38e17-5eaf-41dd-8210-35fa63aeee06 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6bd23ad8458f40828b8071193b91a99d', 'request_id': 'req-8dc38e17-5eaf-41dd-8210-35fa63aeee06'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:49:04.216631 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-8dc38e17-5eaf-41dd-8210-35fa63aeee06 admin admin] The token cache is being invalidate because project 6bd23ad8458f40828b8071193b91a99d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87923) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:49:04.216919 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-8dc38e17-5eaf-41dd-8210-35fa63aeee06 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:49:04.217340 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-8dc38e17-5eaf-41dd-8210-35fa63aeee06 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:49:04.218709 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 748/1510] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:49:03 2022] DELETE /identity/v3/projects/6bd23ad8458f40828b8071193b91a99d => generated 0 bytes in 235 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:49:13.655977 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=62 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:49:14.214573 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=84,keystone:DELETE=7,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:50:35.265766 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9935b92b-c7be-4160-a421-2bc621a4e3a0 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:50:35.266375 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9935b92b-c7be-4160-a421-2bc621a4e3a0 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:50:35.266375 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9935b92b-c7be-4160-a421-2bc621a4e3a0 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:50:35.268440 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 763/1511] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 11 00:50:35 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:50:35.273298 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44a5c44-5251-4572-b9f5-f24061b092ab None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:50:35.273740 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44a5c44-5251-4572-b9f5-f24061b092ab None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:50:35.273847 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44a5c44-5251-4572-b9f5-f24061b092ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:50:35.355438 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-c44a5c44-5251-4572-b9f5-f24061b092ab None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:50:35.356558 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-c44a5c44-5251-4572-b9f5-f24061b092ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:50:35.458740 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 749/1512] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 11 00:50:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:50:35.611609 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9935b92b-c7be-4160-a421-2bc621a4e3a0 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:50:35.613728 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-9935b92b-c7be-4160-a421-2bc621a4e3a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:50:35.731590 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9935b92b-c7be-4160-a421-2bc621a4e3a0 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:50:35.737618 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22f0368b-3111-4cd7-8f62-203dbc476d1c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '835def0e6c6d4365a2767051509eb0d6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'ResellerAdmin', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:50:35.738219 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f0368b-3111-4cd7-8f62-203dbc476d1c service nova] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:50:35.738499 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f0368b-3111-4cd7-8f62-203dbc476d1c service nova] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:50:35.738767 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f0368b-3111-4cd7-8f62-203dbc476d1c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:50:35.746734 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22f0368b-3111-4cd7-8f62-203dbc476d1c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:50:35.748345 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22f0368b-3111-4cd7-8f62-203dbc476d1c service nova] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:50:35.758148 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 764/1513] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:50:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:50:36.635832 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543a44ee-5c4d-4602-aa55-a808abbd455b None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:50:36.636412 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543a44ee-5c4d-4602-aa55-a808abbd455b None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:50:36.636412 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543a44ee-5c4d-4602-aa55-a808abbd455b None None] PATH_INFO: `/v3` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:50:36.637662 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 750/1514] 10.222.0.18 () {54 vars in 925 bytes} [Sun Sep 11 00:50:36 2022] GET /identity/v3 => generated 255 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:50:36.672257 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a022434-933a-4fd2-86c2-414ae5df04f1 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:50:36.672759 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a022434-933a-4fd2-86c2-414ae5df04f1 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:50:36.672850 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a022434-933a-4fd2-86c2-414ae5df04f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:50:36.739158 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-0a022434-933a-4fd2-86c2-414ae5df04f1 None None] MFA Rules not processed for user `cdbfc2ccd73e4a5eb8c999be5b094619`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:50:36.740392 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-0a022434-933a-4fd2-86c2-414ae5df04f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:50:36.809431 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:50:36.816167 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 765/1515] 10.222.0.18 () {58 vars in 1015 bytes} [Sun Sep 11 00:50:36 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:50:36.849430 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062b54e1-f9fe-4fdb-ae0e-9654d2601cc8 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:50:36.849716 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062b54e1-f9fe-4fdb-ae0e-9654d2601cc8 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:50:36.849988 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062b54e1-f9fe-4fdb-ae0e-9654d2601cc8 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:50:36.851371 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 751/1516] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 11 00:50:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:50:36.857724 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d907d3af-3089-4119-88e3-f77a6a76ab6a None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:50:36.858012 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d907d3af-3089-4119-88e3-f77a6a76ab6a None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:50:36.858291 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d907d3af-3089-4119-88e3-f77a6a76ab6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:50:36.878209 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-d907d3af-3089-4119-88e3-f77a6a76ab6a None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:50:36.879445 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-d907d3af-3089-4119-88e3-f77a6a76ab6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:50:36.887375 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 766/1517] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 11 00:50:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:50:37.039098 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062b54e1-f9fe-4fdb-ae0e-9654d2601cc8 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:50:37.054109 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062b54e1-f9fe-4fdb-ae0e-9654d2601cc8 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:50:37.059460 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64addf23-da52-4edb-8775-32a4c524acf2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '835def0e6c6d4365a2767051509eb0d6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'ResellerAdmin', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:50:37.060043 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64addf23-da52-4edb-8775-32a4c524acf2 service nova] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:50:37.060315 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64addf23-da52-4edb-8775-32a4c524acf2 service nova] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:50:37.060580 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64addf23-da52-4edb-8775-32a4c524acf2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:50:37.069689 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64addf23-da52-4edb-8775-32a4c524acf2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:50:37.072699 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64addf23-da52-4edb-8775-32a4c524acf2 service nova] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:50:37.083971 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 752/1518] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:50:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:50:45.379406 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e423126-21f1-4c28-b843-3a78ca84ec02 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:50:45.379860 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e423126-21f1-4c28-b843-3a78ca84ec02 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:50:45.379968 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e423126-21f1-4c28-b843-3a78ca84ec02 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:50:45.381273 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 767/1519] 10.222.0.27 () {54 vars in 961 bytes} [Sun Sep 11 00:50:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:50:45.386319 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4579ed67-bd75-4789-9e4b-213f00b1f390 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:50:45.386740 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4579ed67-bd75-4789-9e4b-213f00b1f390 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:50:45.386949 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4579ed67-bd75-4789-9e4b-213f00b1f390 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:50:45.406964 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-4579ed67-bd75-4789-9e4b-213f00b1f390 None None] MFA Rules not processed for user `cdbfc2ccd73e4a5eb8c999be5b094619`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:50:45.408168 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-4579ed67-bd75-4789-9e4b-213f00b1f390 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:50:45.417108 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 753/1520] 10.222.0.27 () {58 vars in 1058 bytes} [Sun Sep 11 00:50:45 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:50:45.424981 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e423126-21f1-4c28-b843-3a78ca84ec02 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:50:45.438706 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e423126-21f1-4c28-b843-3a78ca84ec02 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:50:45.444565 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b0dedc-09c7-41b0-b96f-672b44cb5658 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cdbfc2ccd73e4a5eb8c999be5b094619', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:50:45.445153 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b0dedc-09c7-41b0-b96f-672b44cb5658 service placement] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:50:45.445425 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b0dedc-09c7-41b0-b96f-672b44cb5658 service placement] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:50:45.445691 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b0dedc-09c7-41b0-b96f-672b44cb5658 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:50:45.453887 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b0dedc-09c7-41b0-b96f-672b44cb5658 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:50:45.455488 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b0dedc-09c7-41b0-b96f-672b44cb5658 service placement] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:50:45.465069 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 768/1521] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:50:45 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 11 00:50:48.576050 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eef8d70-d2eb-4e94-82ee-ec6958b7e586 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:50:48.576675 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eef8d70-d2eb-4e94-82ee-ec6958b7e586 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:50:48.576675 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eef8d70-d2eb-4e94-82ee-ec6958b7e586 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:50:48.577999 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 754/1522] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 11 00:50:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:50:48.584623 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b041961-2425-4e58-bbe4-4fc78ccc5af9 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:50:48.585053 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b041961-2425-4e58-bbe4-4fc78ccc5af9 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:50:48.585180 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b041961-2425-4e58-bbe4-4fc78ccc5af9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:50:48.605270 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-9b041961-2425-4e58-bbe4-4fc78ccc5af9 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:50:48.606460 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-9b041961-2425-4e58-bbe4-4fc78ccc5af9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:50:48.614913 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 769/1523] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 11 00:50:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:50:48.769122 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eef8d70-d2eb-4e94-82ee-ec6958b7e586 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:50:48.783425 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eef8d70-d2eb-4e94-82ee-ec6958b7e586 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:50:48.789825 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddaea0f8-533a-4224-9924-aa0e69f7172d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '835def0e6c6d4365a2767051509eb0d6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'ResellerAdmin', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:50:48.790414 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddaea0f8-533a-4224-9924-aa0e69f7172d service nova] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:50:48.790693 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddaea0f8-533a-4224-9924-aa0e69f7172d service nova] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:50:48.790997 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddaea0f8-533a-4224-9924-aa0e69f7172d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:50:48.798919 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddaea0f8-533a-4224-9924-aa0e69f7172d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:50:48.801181 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddaea0f8-533a-4224-9924-aa0e69f7172d service nova] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:50:48.810389 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 755/1524] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:50:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:50:49.881323 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64b12e8-b0f2-4ccb-b134-8dab8189bc00 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:50:49.881867 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64b12e8-b0f2-4ccb-b134-8dab8189bc00 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:50:49.881867 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64b12e8-b0f2-4ccb-b134-8dab8189bc00 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:50:49.883233 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 770/1525] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 11 00:50:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:50:49.889460 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04377115-1974-4021-b0db-7376ca06f816 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:50:49.889876 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04377115-1974-4021-b0db-7376ca06f816 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:50:49.890016 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04377115-1974-4021-b0db-7376ca06f816 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:50:49.910766 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-04377115-1974-4021-b0db-7376ca06f816 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:50:49.912015 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-04377115-1974-4021-b0db-7376ca06f816 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:50:49.919756 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 756/1526] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 11 00:50:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:50:50.069148 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b64b12e8-b0f2-4ccb-b134-8dab8189bc00 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:50:50.084154 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b64b12e8-b0f2-4ccb-b134-8dab8189bc00 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:50:50.089701 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb249fed-0b37-4534-a5dc-ea81ea7e76c0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '835def0e6c6d4365a2767051509eb0d6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'ResellerAdmin', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:50:50.090297 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb249fed-0b37-4534-a5dc-ea81ea7e76c0 service nova] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:50:50.090571 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb249fed-0b37-4534-a5dc-ea81ea7e76c0 service nova] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:50:50.090885 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb249fed-0b37-4534-a5dc-ea81ea7e76c0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:50:50.099769 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb249fed-0b37-4534-a5dc-ea81ea7e76c0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:50:50.101428 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb249fed-0b37-4534-a5dc-ea81ea7e76c0 service nova] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:50:50.111153 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 771/1527] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:50:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 11 00:50:54.185464 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17fb2b35-fb1e-4f49-bd87-7c77cf4e7032 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:50:54.186009 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17fb2b35-fb1e-4f49-bd87-7c77cf4e7032 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:50:54.186009 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17fb2b35-fb1e-4f49-bd87-7c77cf4e7032 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:50:54.187379 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 757/1528] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 11 00:50:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:50:54.192509 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8b2440-a294-409f-a796-c691b99aa3a0 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:50:54.192951 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8b2440-a294-409f-a796-c691b99aa3a0 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:50:54.193050 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8b2440-a294-409f-a796-c691b99aa3a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:50:54.212342 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-ce8b2440-a294-409f-a796-c691b99aa3a0 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:50:54.213544 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-ce8b2440-a294-409f-a796-c691b99aa3a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:50:54.221759 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 772/1529] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 11 00:50:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:50:54.511167 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17fb2b35-fb1e-4f49-bd87-7c77cf4e7032 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:50:54.512864 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-17fb2b35-fb1e-4f49-bd87-7c77cf4e7032 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:50:54.547055 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:50:54.595794 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17fb2b35-fb1e-4f49-bd87-7c77cf4e7032 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:50:54.600355 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b797b05-5aa6-4222-8c3d-27dd91fbe8f7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:50:54.600966 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b797b05-5aa6-4222-8c3d-27dd91fbe8f7 service neutron] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:50:54.601242 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b797b05-5aa6-4222-8c3d-27dd91fbe8f7 service neutron] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:50:54.601512 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b797b05-5aa6-4222-8c3d-27dd91fbe8f7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:50:54.610720 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b797b05-5aa6-4222-8c3d-27dd91fbe8f7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:50:54.612614 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b797b05-5aa6-4222-8c3d-27dd91fbe8f7 service neutron] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:50:54.622017 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 758/1530] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:50:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:50:56.770438 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fcbfae-475c-4f74-8d95-99ac5eb91de3 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:50:56.771012 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fcbfae-475c-4f74-8d95-99ac5eb91de3 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:50:56.771012 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fcbfae-475c-4f74-8d95-99ac5eb91de3 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:50:56.772425 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 773/1531] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 11 00:50:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:50:56.777569 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-212be720-3e48-49cd-a364-d3da89976756 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:50:56.778019 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-212be720-3e48-49cd-a364-d3da89976756 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:50:56.778119 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-212be720-3e48-49cd-a364-d3da89976756 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:50:56.797994 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-212be720-3e48-49cd-a364-d3da89976756 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:50:56.800649 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-212be720-3e48-49cd-a364-d3da89976756 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:50:56.808462 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 759/1532] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 11 00:50:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:50:57.093300 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12fcbfae-475c-4f74-8d95-99ac5eb91de3 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:50:57.108059 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12fcbfae-475c-4f74-8d95-99ac5eb91de3 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:50:57.112271 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-203e0af8-b502-4c68-8474-014f3ce44099 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:50:57.112890 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203e0af8-b502-4c68-8474-014f3ce44099 service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:50:57.113167 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203e0af8-b502-4c68-8474-014f3ce44099 service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:50:57.113436 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203e0af8-b502-4c68-8474-014f3ce44099 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:50:57.122225 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-203e0af8-b502-4c68-8474-014f3ce44099 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:50:57.123679 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-203e0af8-b502-4c68-8474-014f3ce44099 service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:50:57.132844 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 774/1533] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:50:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:51:01.148003 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8161c397-effe-4410-8b1f-d089e05760eb None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:51:01.148560 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8161c397-effe-4410-8b1f-d089e05760eb None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:51:01.148560 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8161c397-effe-4410-8b1f-d089e05760eb None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:51:01.149881 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 760/1534] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 11 00:51:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 11 00:51:01.155439 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9d1701-22d3-4e7f-9463-5e873ff0dfb6 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:51:01.155885 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9d1701-22d3-4e7f-9463-5e873ff0dfb6 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:51:01.155994 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9d1701-22d3-4e7f-9463-5e873ff0dfb6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:51:01.175931 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-0a9d1701-22d3-4e7f-9463-5e873ff0dfb6 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:51:01.177142 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-0a9d1701-22d3-4e7f-9463-5e873ff0dfb6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:51:01.184894 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 775/1535] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 11 00:51:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:51:01.487234 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8161c397-effe-4410-8b1f-d089e05760eb None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:51:01.488886 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-8161c397-effe-4410-8b1f-d089e05760eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:51:01.567310 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8161c397-effe-4410-8b1f-d089e05760eb None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:51:01.570892 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c1ebb9-0c34-41d3-a2e6-a377d0d0671f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:51:01.571493 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c1ebb9-0c34-41d3-a2e6-a377d0d0671f service cinder] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:51:01.571771 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c1ebb9-0c34-41d3-a2e6-a377d0d0671f service cinder] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:51:01.572038 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c1ebb9-0c34-41d3-a2e6-a377d0d0671f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:51:01.580937 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04c1ebb9-0c34-41d3-a2e6-a377d0d0671f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:51:01.582360 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04c1ebb9-0c34-41d3-a2e6-a377d0d0671f service cinder] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:51:01.591603 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 761/1536] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:51:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:51:03.733427 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3875ceb1-bcb1-466d-b0af-d715ae30e34d None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:51:03.733960 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3875ceb1-bcb1-466d-b0af-d715ae30e34d None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:51:03.733960 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3875ceb1-bcb1-466d-b0af-d715ae30e34d None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:51:03.735340 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 776/1537] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 11 00:51:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:51:03.740601 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e24f052-6b4f-4014-b097-c0803722d518 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:51:03.741000 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e24f052-6b4f-4014-b097-c0803722d518 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:51:03.741153 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e24f052-6b4f-4014-b097-c0803722d518 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:51:03.761089 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-5e24f052-6b4f-4014-b097-c0803722d518 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:51:03.762280 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-5e24f052-6b4f-4014-b097-c0803722d518 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:51:03.771103 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 762/1538] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 11 00:51:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:51:04.073179 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3875ceb1-bcb1-466d-b0af-d715ae30e34d None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:51:04.074882 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-3875ceb1-bcb1-466d-b0af-d715ae30e34d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:51:04.088134 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3875ceb1-bcb1-466d-b0af-d715ae30e34d None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:51:04.091623 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c510d47-9c5b-48c3-a074-d13f4b1a1dd2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:51:04.092212 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c510d47-9c5b-48c3-a074-d13f4b1a1dd2 service cinder] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:51:04.092488 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c510d47-9c5b-48c3-a074-d13f4b1a1dd2 service cinder] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:51:04.092759 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c510d47-9c5b-48c3-a074-d13f4b1a1dd2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:51:04.102746 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c510d47-9c5b-48c3-a074-d13f4b1a1dd2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:51:04.104244 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c510d47-9c5b-48c3-a074-d13f4b1a1dd2 service cinder] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:51:04.114463 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 777/1539] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:51:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:51:13.756175 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=63 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:51:14.109520 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=37 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:52:28.728997 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14132268-ba3b-4204-859b-b0cb2f8c9a41 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:28.740000 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14132268-ba3b-4204-859b-b0cb2f8c9a41 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:28.740000 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14132268-ba3b-4204-859b-b0cb2f8c9a41 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:28.760707 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-14132268-ba3b-4204-859b-b0cb2f8c9a41 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:28.762443 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-14132268-ba3b-4204-859b-b0cb2f8c9a41 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:28.776258 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 763/1540] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:28.786885 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9946459-e7d4-4d78-a23f-a7dd801e86b0 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:28.788060 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9946459-e7d4-4d78-a23f-a7dd801e86b0 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:28.788584 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9946459-e7d4-4d78-a23f-a7dd801e86b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:28.819966 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-d9946459-e7d4-4d78-a23f-a7dd801e86b0 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:28.821593 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-d9946459-e7d4-4d78-a23f-a7dd801e86b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:28.836294 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 778/1541] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:28.841402 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14132268-ba3b-4204-859b-b0cb2f8c9a41 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:28.856150 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14132268-ba3b-4204-859b-b0cb2f8c9a41 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:28.864002 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e96ebe12-ee93-4039-a35b-bbdbbef2cf74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:28.864749 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96ebe12-ee93-4039-a35b-bbdbbef2cf74 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:28.865030 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96ebe12-ee93-4039-a35b-bbdbbef2cf74 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:28.865300 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96ebe12-ee93-4039-a35b-bbdbbef2cf74 admin admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:28.865954 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e96ebe12-ee93-4039-a35b-bbdbbef2cf74 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:52:28.866253 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e96ebe12-ee93-4039-a35b-bbdbbef2cf74 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:28.867882 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e96ebe12-ee93-4039-a35b-bbdbbef2cf74 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:28.881814 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7546341b-0f80-4fdc-a5f4-a8e982ad3e7e None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:28.882458 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7546341b-0f80-4fdc-a5f4-a8e982ad3e7e None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:28.882984 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7546341b-0f80-4fdc-a5f4-a8e982ad3e7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:28.890898 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 764/1542] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 00:52:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:52:28.916212 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e96ebe12-ee93-4039-a35b-bbdbbef2cf74 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:28.916455 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-7546341b-0f80-4fdc-a5f4-a8e982ad3e7e None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:28.918033 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-7546341b-0f80-4fdc-a5f4-a8e982ad3e7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:28.929648 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e96ebe12-ee93-4039-a35b-bbdbbef2cf74 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:28.931715 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 779/1543] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:28.940361 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1044aa-8a6f-40b5-a672-3afa28915c3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:28.940991 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1044aa-8a6f-40b5-a672-3afa28915c3a admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:28.941267 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1044aa-8a6f-40b5-a672-3afa28915c3a admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:28.941536 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1044aa-8a6f-40b5-a672-3afa28915c3a admin admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:28.942189 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1044aa-8a6f-40b5-a672-3afa28915c3a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:28.942643 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98be3883-bf17-438f-9bb7-fc4a392047b6 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:28.943178 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98be3883-bf17-438f-9bb7-fc4a392047b6 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:28.943312 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: WARNING py.warnings [None req-ed1044aa-8a6f-40b5-a672-3afa28915c3a admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 11 00:52:28.943312 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: warnings.warn(msg) Sep 11 00:52:28.943312 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: Sep 11 00:52:28.943969 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98be3883-bf17-438f-9bb7-fc4a392047b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:28.944444 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1044aa-8a6f-40b5-a672-3afa28915c3a admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:28.975991 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-98be3883-bf17-438f-9bb7-fc4a392047b6 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:28.977708 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-98be3883-bf17-438f-9bb7-fc4a392047b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:28.993085 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 780/1544] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 52 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:28.997581 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98be3883-bf17-438f-9bb7-fc4a392047b6 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:29.012530 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98be3883-bf17-438f-9bb7-fc4a392047b6 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:29.023887 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-733e75ad-2072-487f-8dd3-fc77c0dc468d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:29.024886 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733e75ad-2072-487f-8dd3-fc77c0dc468d admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:29.025380 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733e75ad-2072-487f-8dd3-fc77c0dc468d admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:29.026059 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733e75ad-2072-487f-8dd3-fc77c0dc468d admin admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:29.027043 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-733e75ad-2072-487f-8dd3-fc77c0dc468d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:52:29.027579 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-733e75ad-2072-487f-8dd3-fc77c0dc468d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:29.028830 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: WARNING py.warnings [None req-733e75ad-2072-487f-8dd3-fc77c0dc468d admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 11 00:52:29.028830 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: warnings.warn(msg) Sep 11 00:52:29.028830 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: Sep 11 00:52:29.030202 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-733e75ad-2072-487f-8dd3-fc77c0dc468d admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:29.037049 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 765/1545] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:52:28 2022] POST /identity/v3/projects => generated 367 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:52:29.037049 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8713c6b-9c99-4a7b-9e59-99e61e907267 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:29.037049 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8713c6b-9c99-4a7b-9e59-99e61e907267 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:29.037049 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8713c6b-9c99-4a7b-9e59-99e61e907267 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:29.053944 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 781/1546] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 00:52:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:52:29.055208 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-733e75ad-2072-487f-8dd3-fc77c0dc468d admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:29.069079 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-733e75ad-2072-487f-8dd3-fc77c0dc468d admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:29.076752 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-c8713c6b-9c99-4a7b-9e59-99e61e907267 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:29.078900 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-c8713c6b-9c99-4a7b-9e59-99e61e907267 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:29.169947 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee9cd737-45ba-4761-a130-05a5c17a9325 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:29.170562 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9cd737-45ba-4761-a130-05a5c17a9325 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:29.170871 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9cd737-45ba-4761-a130-05a5c17a9325 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:29.171147 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9cd737-45ba-4761-a130-05a5c17a9325 admin admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:29.171833 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee9cd737-45ba-4761-a130-05a5c17a9325 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:29.173004 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: WARNING py.warnings [None req-ee9cd737-45ba-4761-a130-05a5c17a9325 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 11 00:52:29.173004 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: warnings.warn(msg) Sep 11 00:52:29.173004 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: Sep 11 00:52:29.174239 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee9cd737-45ba-4761-a130-05a5c17a9325 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:29.186882 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 766/1547] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:29.187859 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8713c6b-9c99-4a7b-9e59-99e61e907267 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:29.197434 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 782/1548] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:52:29 2022] POST /identity/v3/users => generated 284 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:52:29.198810 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab618a66-5646-4c9f-98d2-88319bce44bf None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:29.198981 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8713c6b-9c99-4a7b-9e59-99e61e907267 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:29.199105 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab618a66-5646-4c9f-98d2-88319bce44bf None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:29.199381 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab618a66-5646-4c9f-98d2-88319bce44bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:29.204464 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c803fc69-1c3b-4080-93a4-110e62d57079 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:29.205058 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c803fc69-1c3b-4080-93a4-110e62d57079 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:29.205334 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c803fc69-1c3b-4080-93a4-110e62d57079 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:29.205603 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c803fc69-1c3b-4080-93a4-110e62d57079 admin admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:29.206227 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c803fc69-1c3b-4080-93a4-110e62d57079 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:29.208159 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c803fc69-1c3b-4080-93a4-110e62d57079 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:29.220221 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-ab618a66-5646-4c9f-98d2-88319bce44bf None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:29.221430 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-ab618a66-5646-4c9f-98d2-88319bce44bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:29.230642 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 783/1549] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:29.236333 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c808a1-ab97-4e7d-8016-6691a85fc422 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:29.236898 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c808a1-ab97-4e7d-8016-6691a85fc422 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:29.237348 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c808a1-ab97-4e7d-8016-6691a85fc422 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:29.248461 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 767/1550] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:52:29 2022] POST /identity/v3/projects => generated 387 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:52:29.248461 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1d13f3-f50c-407a-916a-8f279bc68240 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:29.248461 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1d13f3-f50c-407a-916a-8f279bc68240 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:29.248461 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1d13f3-f50c-407a-916a-8f279bc68240 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:29.261327 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-81c808a1-ab97-4e7d-8016-6691a85fc422 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:29.261518 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-ed1d13f3-f50c-407a-916a-8f279bc68240 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:29.262536 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-81c808a1-ab97-4e7d-8016-6691a85fc422 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:29.262663 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-ed1d13f3-f50c-407a-916a-8f279bc68240 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:29.322602 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 784/1551] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:29.331212 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2b082d-e507-4d75-a538-a75916400043 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:29.331432 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 768/1552] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:29.331540 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2b082d-e507-4d75-a538-a75916400043 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:29.331540 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2b082d-e507-4d75-a538-a75916400043 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:29.331782 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09de6e56-6ec9-4004-bc2c-fb9cbd38914a None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:29.331782 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09de6e56-6ec9-4004-bc2c-fb9cbd38914a None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:29.331782 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09de6e56-6ec9-4004-bc2c-fb9cbd38914a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:29.348593 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-09de6e56-6ec9-4004-bc2c-fb9cbd38914a None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:29.348815 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-fe2b082d-e507-4d75-a538-a75916400043 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:29.349796 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-09de6e56-6ec9-4004-bc2c-fb9cbd38914a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:29.349927 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-fe2b082d-e507-4d75-a538-a75916400043 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:29.357889 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 785/1553] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:29.358035 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 769/1554] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:29.359030 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0fa4ca0-3cae-458e-a5ce-4bc402243c9c None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:29.359519 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0fa4ca0-3cae-458e-a5ce-4bc402243c9c None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:29.359962 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0fa4ca0-3cae-458e-a5ce-4bc402243c9c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:29.360261 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09de6e56-6ec9-4004-bc2c-fb9cbd38914a None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:29.374274 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09de6e56-6ec9-4004-bc2c-fb9cbd38914a None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:29.379694 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7525c382-a2a5-43c1-b494-79d664444019 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:29.380236 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-c0fa4ca0-3cae-458e-a5ce-4bc402243c9c None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:29.380369 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7525c382-a2a5-43c1-b494-79d664444019 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:29.380558 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7525c382-a2a5-43c1-b494-79d664444019 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:29.380830 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7525c382-a2a5-43c1-b494-79d664444019 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:29.381407 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-c0fa4ca0-3cae-458e-a5ce-4bc402243c9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:29.381521 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7525c382-a2a5-43c1-b494-79d664444019 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:29.383126 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7525c382-a2a5-43c1-b494-79d664444019 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:29.389391 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 786/1555] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:29.391629 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aef4f98c-d168-4e41-8966-f7f3d8fd3263 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:29.391951 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aef4f98c-d168-4e41-8966-f7f3d8fd3263 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:29.392225 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aef4f98c-d168-4e41-8966-f7f3d8fd3263 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:29.397087 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 770/1556] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:52:29 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:52:29.398074 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7525c382-a2a5-43c1-b494-79d664444019 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:29.409357 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7525c382-a2a5-43c1-b494-79d664444019 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:29.414234 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a24baca-4a31-4d55-baf3-1d6eab2f3f1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:29.414371 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-aef4f98c-d168-4e41-8966-f7f3d8fd3263 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:29.414848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a24baca-4a31-4d55-baf3-1d6eab2f3f1b admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:29.415132 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a24baca-4a31-4d55-baf3-1d6eab2f3f1b admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:29.415409 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a24baca-4a31-4d55-baf3-1d6eab2f3f1b admin admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:29.415517 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-aef4f98c-d168-4e41-8966-f7f3d8fd3263 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:29.416037 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a24baca-4a31-4d55-baf3-1d6eab2f3f1b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:29.418100 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a24baca-4a31-4d55-baf3-1d6eab2f3f1b admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:29.423683 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 787/1557] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:29.424761 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5214b0d4-adb1-4045-bbcd-9c5d9e80b529 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:29.425050 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5214b0d4-adb1-4045-bbcd-9c5d9e80b529 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:29.425320 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5214b0d4-adb1-4045-bbcd-9c5d9e80b529 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:29.428590 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:52:29.445795 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 771/1558] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:52:29 2022] POST /identity/v3/users => generated 294 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:52:29.445795 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d521469f-0259-40ac-9f56-948790768d3b None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:29.445795 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d521469f-0259-40ac-9f56-948790768d3b None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:29.445795 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d521469f-0259-40ac-9f56-948790768d3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:29.446251 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-5214b0d4-adb1-4045-bbcd-9c5d9e80b529 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:29.447254 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-5214b0d4-adb1-4045-bbcd-9c5d9e80b529 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:29.455873 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 788/1559] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:29.457052 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903ab893-9699-44fa-bb83-deb8bce6cdfc None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:29.457336 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903ab893-9699-44fa-bb83-deb8bce6cdfc None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:29.457608 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903ab893-9699-44fa-bb83-deb8bce6cdfc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:29.465757 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-d521469f-0259-40ac-9f56-948790768d3b None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:29.466987 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-d521469f-0259-40ac-9f56-948790768d3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:29.475074 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 772/1560] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:29.476254 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af267af0-c80f-42c4-b6b0-ff9da250440b None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:29.476536 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af267af0-c80f-42c4-b6b0-ff9da250440b None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:29.476809 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af267af0-c80f-42c4-b6b0-ff9da250440b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:29.479793 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-903ab893-9699-44fa-bb83-deb8bce6cdfc None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:29.480981 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-903ab893-9699-44fa-bb83-deb8bce6cdfc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:29.489338 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 789/1561] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:29.490172 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-903ab893-9699-44fa-bb83-deb8bce6cdfc None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:29.497764 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-af267af0-c80f-42c4-b6b0-ff9da250440b None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:29.499031 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-af267af0-c80f-42c4-b6b0-ff9da250440b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:29.500329 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-903ab893-9699-44fa-bb83-deb8bce6cdfc None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:29.504903 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b1f9f58-7d0e-48fd-a9cc-e5cfe7b27ff8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:29.505487 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1f9f58-7d0e-48fd-a9cc-e5cfe7b27ff8 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:29.505766 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1f9f58-7d0e-48fd-a9cc-e5cfe7b27ff8 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:29.506035 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1f9f58-7d0e-48fd-a9cc-e5cfe7b27ff8 admin admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:29.506673 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b1f9f58-7d0e-48fd-a9cc-e5cfe7b27ff8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:52:29.507012 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b1f9f58-7d0e-48fd-a9cc-e5cfe7b27ff8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:29.507590 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 773/1562] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:29.508422 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af267af0-c80f-42c4-b6b0-ff9da250440b None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:29.508605 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b1f9f58-7d0e-48fd-a9cc-e5cfe7b27ff8 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:29.521207 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af267af0-c80f-42c4-b6b0-ff9da250440b None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:29.522116 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 790/1563] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 00:52:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:52:29.523256 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b1f9f58-7d0e-48fd-a9cc-e5cfe7b27ff8 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:29.526888 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff9b58f-036f-4a44-a969-a0ec77b64fa1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:29.527590 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff9b58f-036f-4a44-a969-a0ec77b64fa1 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:29.528027 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff9b58f-036f-4a44-a969-a0ec77b64fa1 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:29.528330 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff9b58f-036f-4a44-a969-a0ec77b64fa1 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:29.529624 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ff9b58f-036f-4a44-a969-a0ec77b64fa1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:29.529765 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:52:29.531221 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ff9b58f-036f-4a44-a969-a0ec77b64fa1 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:29.534087 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b1f9f58-7d0e-48fd-a9cc-e5cfe7b27ff8 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:29.539947 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b7425e3-dc1f-403d-b9ff-c72c1b9b11d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:29.540590 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7425e3-dc1f-403d-b9ff-c72c1b9b11d4 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:29.540871 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7425e3-dc1f-403d-b9ff-c72c1b9b11d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:29.541144 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7425e3-dc1f-403d-b9ff-c72c1b9b11d4 admin admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:29.541781 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b7425e3-dc1f-403d-b9ff-c72c1b9b11d4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:52:29.542083 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b7425e3-dc1f-403d-b9ff-c72c1b9b11d4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:29.542213 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 774/1564] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:52:29 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:52:29.543854 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b7425e3-dc1f-403d-b9ff-c72c1b9b11d4 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:29.546227 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff9b58f-036f-4a44-a969-a0ec77b64fa1 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:29.556136 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 791/1565] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 00:52:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:52:29.556738 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff9b58f-036f-4a44-a969-a0ec77b64fa1 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:29.556994 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b7425e3-dc1f-403d-b9ff-c72c1b9b11d4 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:29.561471 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc813b99-56b0-4f01-82ea-e25f28c2d785 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:29.562615 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc813b99-56b0-4f01-82ea-e25f28c2d785 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:29.562922 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc813b99-56b0-4f01-82ea-e25f28c2d785 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:29.563201 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc813b99-56b0-4f01-82ea-e25f28c2d785 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:29.563816 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc813b99-56b0-4f01-82ea-e25f28c2d785 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:29.565432 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc813b99-56b0-4f01-82ea-e25f28c2d785 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:29.569556 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b7425e3-dc1f-403d-b9ff-c72c1b9b11d4 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:29.574347 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3e1a195-4d1e-422b-b6c0-cc9efeb08e65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:29.574975 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e1a195-4d1e-422b-b6c0-cc9efeb08e65 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:29.575260 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e1a195-4d1e-422b-b6c0-cc9efeb08e65 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:29.575532 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e1a195-4d1e-422b-b6c0-cc9efeb08e65 admin admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:29.575971 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 775/1566] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:52:29 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:52:29.576172 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3e1a195-4d1e-422b-b6c0-cc9efeb08e65 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:52:29.576541 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3e1a195-4d1e-422b-b6c0-cc9efeb08e65 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:29.576650 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc813b99-56b0-4f01-82ea-e25f28c2d785 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:29.578826 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3e1a195-4d1e-422b-b6c0-cc9efeb08e65 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:29.586553 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc813b99-56b0-4f01-82ea-e25f28c2d785 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:29.597226 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 792/1567] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 00:52:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:52:29.597399 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec79587-aac1-4fbf-b971-4daa780d35e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:29.597517 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3e1a195-4d1e-422b-b6c0-cc9efeb08e65 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:29.597608 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec79587-aac1-4fbf-b971-4daa780d35e1 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:29.597608 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec79587-aac1-4fbf-b971-4daa780d35e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:29.597608 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec79587-aac1-4fbf-b971-4daa780d35e1 admin admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:29.597608 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ec79587-aac1-4fbf-b971-4daa780d35e1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:52:29.597608 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ec79587-aac1-4fbf-b971-4daa780d35e1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:29.597608 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ec79587-aac1-4fbf-b971-4daa780d35e1 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:29.603029 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3e1a195-4d1e-422b-b6c0-cc9efeb08e65 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:29.607621 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb5d34c-24cd-444e-b8c7-6d0fff0b04b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:29.608186 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 776/1568] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 00:52:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:52:29.608302 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb5d34c-24cd-444e-b8c7-6d0fff0b04b3 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:29.608475 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb5d34c-24cd-444e-b8c7-6d0fff0b04b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:29.609287 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb5d34c-24cd-444e-b8c7-6d0fff0b04b3 admin admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:29.609583 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec79587-aac1-4fbf-b971-4daa780d35e1 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:29.610205 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb5d34c-24cd-444e-b8c7-6d0fff0b04b3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:52:29.610520 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb5d34c-24cd-444e-b8c7-6d0fff0b04b3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:29.612161 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb5d34c-24cd-444e-b8c7-6d0fff0b04b3 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:29.622709 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec79587-aac1-4fbf-b971-4daa780d35e1 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:29.632207 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 793/1569] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 00:52:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:52:29.632207 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb5d34c-24cd-444e-b8c7-6d0fff0b04b3 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:29.632447 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f23ed8f2-7501-4952-864a-a4980e56976e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:29.632447 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23ed8f2-7501-4952-864a-a4980e56976e admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:29.632447 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23ed8f2-7501-4952-864a-a4980e56976e admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:29.632447 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23ed8f2-7501-4952-864a-a4980e56976e admin admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:29.632858 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f23ed8f2-7501-4952-864a-a4980e56976e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:52:29.632858 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f23ed8f2-7501-4952-864a-a4980e56976e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:29.632858 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f23ed8f2-7501-4952-864a-a4980e56976e admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:29.636136 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb5d34c-24cd-444e-b8c7-6d0fff0b04b3 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:29.640689 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d88a4dfb-d152-4fce-8c87-6f9c1ae0bbdb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:29.641264 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88a4dfb-d152-4fce-8c87-6f9c1ae0bbdb admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:29.641539 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88a4dfb-d152-4fce-8c87-6f9c1ae0bbdb admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:29.641807 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88a4dfb-d152-4fce-8c87-6f9c1ae0bbdb admin admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:29.642428 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d88a4dfb-d152-4fce-8c87-6f9c1ae0bbdb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:29.644153 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 777/1570] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 00:52:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:52:29.644405 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d88a4dfb-d152-4fce-8c87-6f9c1ae0bbdb admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:29.644855 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f23ed8f2-7501-4952-864a-a4980e56976e admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:29.656192 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f23ed8f2-7501-4952-864a-a4980e56976e admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:29.661041 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a4ab3e-e5f8-4fbe-8e50-87ac91a39ba1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:29.661680 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a4ab3e-e5f8-4fbe-8e50-87ac91a39ba1 admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:29.661956 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a4ab3e-e5f8-4fbe-8e50-87ac91a39ba1 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:29.662226 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a4ab3e-e5f8-4fbe-8e50-87ac91a39ba1 admin admin] PATH_INFO: `/v3/projects/93b2f717c20042d0809f2c51a658fb83/users/2b5c14fdcb014dcdae858a902812dd50/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:29.677388 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 794/1571] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:52:29 2022] POST /identity/v3/projects => generated 379 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:52:29.677388 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d88a4dfb-d152-4fce-8c87-6f9c1ae0bbdb admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:29.677686 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43a4ab3e-e5f8-4fbe-8e50-87ac91a39ba1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=93b2f717c20042d0809f2c51a658fb83, user_id=2b5c14fdcb014dcdae858a902812dd50, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:29.678385 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: WARNING py.warnings [None req-43a4ab3e-e5f8-4fbe-8e50-87ac91a39ba1 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 11 00:52:29.678385 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: warnings.warn(msg) Sep 11 00:52:29.678385 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: Sep 11 00:52:29.680920 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43a4ab3e-e5f8-4fbe-8e50-87ac91a39ba1 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:29.695681 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 778/1572] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:52:29 2022] PUT /identity/v3/projects/93b2f717c20042d0809f2c51a658fb83/users/2b5c14fdcb014dcdae858a902812dd50/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:52:29.696494 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a4ab3e-e5f8-4fbe-8e50-87ac91a39ba1 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:29.740036 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d88a4dfb-d152-4fce-8c87-6f9c1ae0bbdb admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:29.769269 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a4ab3e-e5f8-4fbe-8e50-87ac91a39ba1 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:29.775131 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-412bbb29-7611-4f45-bee9-02a713018152 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:29.775741 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412bbb29-7611-4f45-bee9-02a713018152 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:29.776021 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412bbb29-7611-4f45-bee9-02a713018152 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:29.776293 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412bbb29-7611-4f45-bee9-02a713018152 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:29.776936 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-412bbb29-7611-4f45-bee9-02a713018152 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:29.778586 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-412bbb29-7611-4f45-bee9-02a713018152 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:29.792171 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 779/1573] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:52:29 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:52:29.792171 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-412bbb29-7611-4f45-bee9-02a713018152 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:29.797714 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad148e6-dae0-41cb-b355-b69f4730a7ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:29.798358 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad148e6-dae0-41cb-b355-b69f4730a7ba admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:29.798634 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad148e6-dae0-41cb-b355-b69f4730a7ba admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:29.798944 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad148e6-dae0-41cb-b355-b69f4730a7ba admin admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:29.799574 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad148e6-dae0-41cb-b355-b69f4730a7ba admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:29.801477 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad148e6-dae0-41cb-b355-b69f4730a7ba admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:29.802176 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-412bbb29-7611-4f45-bee9-02a713018152 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:29.806891 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8140ffca-e398-4d67-9a63-70a674c80a85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:29.807551 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8140ffca-e398-4d67-9a63-70a674c80a85 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:29.807831 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8140ffca-e398-4d67-9a63-70a674c80a85 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:29.808101 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8140ffca-e398-4d67-9a63-70a674c80a85 admin admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:29.808720 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8140ffca-e398-4d67-9a63-70a674c80a85 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:29.810661 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8140ffca-e398-4d67-9a63-70a674c80a85 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:29.828506 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 795/1574] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:52:29 2022] POST /identity/v3/projects => generated 393 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:52:29.829416 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad148e6-dae0-41cb-b355-b69f4730a7ba admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:29.841029 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 780/1575] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:52:29 2022] POST /identity/v3/projects => generated 363 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:52:29.842089 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8140ffca-e398-4d67-9a63-70a674c80a85 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:29.868123 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:52:29.887051 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:INSERT=3,keystone:SELECT=97 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:52:29.892869 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad148e6-dae0-41cb-b355-b69f4730a7ba admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:29.904868 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8140ffca-e398-4d67-9a63-70a674c80a85 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:29.909992 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04dbcda8-094d-44ba-a691-a77719d2be46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:29.910914 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04dbcda8-094d-44ba-a691-a77719d2be46 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:29.911191 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04dbcda8-094d-44ba-a691-a77719d2be46 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:29.911458 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04dbcda8-094d-44ba-a691-a77719d2be46 admin admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:29.912082 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04dbcda8-094d-44ba-a691-a77719d2be46 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:29.913978 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04dbcda8-094d-44ba-a691-a77719d2be46 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:29.941314 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 781/1576] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:52:29 2022] POST /identity/v3/projects => generated 391 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:52:29.945247 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04dbcda8-094d-44ba-a691-a77719d2be46 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:29.950134 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e4ffe4-2ab1-474d-a374-98cee514a2ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:29.950752 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e4ffe4-2ab1-474d-a374-98cee514a2ad admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:29.951094 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e4ffe4-2ab1-474d-a374-98cee514a2ad admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:29.951500 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e4ffe4-2ab1-474d-a374-98cee514a2ad admin admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:29.952141 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7e4ffe4-2ab1-474d-a374-98cee514a2ad admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:29.954589 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7e4ffe4-2ab1-474d-a374-98cee514a2ad admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:29.985266 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 796/1577] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:52:29 2022] POST /identity/v3/projects => generated 383 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:52:29.985266 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e4ffe4-2ab1-474d-a374-98cee514a2ad admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:30.006590 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04dbcda8-094d-44ba-a691-a77719d2be46 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:30.046880 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e4ffe4-2ab1-474d-a374-98cee514a2ad admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:30.051548 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d13fcd7e-2c71-4b7a-9947-3e06ea872a9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:30.052160 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13fcd7e-2c71-4b7a-9947-3e06ea872a9f admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:30.052444 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13fcd7e-2c71-4b7a-9947-3e06ea872a9f admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:30.052719 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13fcd7e-2c71-4b7a-9947-3e06ea872a9f admin admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:30.053353 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d13fcd7e-2c71-4b7a-9947-3e06ea872a9f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:30.055307 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d13fcd7e-2c71-4b7a-9947-3e06ea872a9f admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:30.063267 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc7b838-dcfc-48dc-8472-8e4f9ae6fb40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:30.063939 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc7b838-dcfc-48dc-8472-8e4f9ae6fb40 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:30.064222 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc7b838-dcfc-48dc-8472-8e4f9ae6fb40 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:30.064487 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc7b838-dcfc-48dc-8472-8e4f9ae6fb40 admin admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:30.065107 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dc7b838-dcfc-48dc-8472-8e4f9ae6fb40 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:30.067006 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dc7b838-dcfc-48dc-8472-8e4f9ae6fb40 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:30.080466 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 797/1578] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:52:29 2022] POST /identity/v3/users => generated 290 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:52:30.080466 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c67559f-3afb-48cb-bade-32870b612563 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:30.080466 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c67559f-3afb-48cb-bade-32870b612563 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:30.080466 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c67559f-3afb-48cb-bade-32870b612563 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:30.096624 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 782/1579] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:52:29 2022] POST /identity/v3/projects => generated 373 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:52:30.097568 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc7b838-dcfc-48dc-8472-8e4f9ae6fb40 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:30.112268 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-3c67559f-3afb-48cb-bade-32870b612563 None None] MFA Rules not processed for user `2b5c14fdcb014dcdae858a902812dd50`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:30.113453 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-3c67559f-3afb-48cb-bade-32870b612563 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:30.157775 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:52:30.163450 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc7b838-dcfc-48dc-8472-8e4f9ae6fb40 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:30.168244 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a29a570-ad7c-40af-aace-fb35ce937c1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:30.168904 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a29a570-ad7c-40af-aace-fb35ce937c1c admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:30.169180 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a29a570-ad7c-40af-aace-fb35ce937c1c admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:30.169449 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a29a570-ad7c-40af-aace-fb35ce937c1c admin admin] PATH_INFO: `/v3/projects/fd9221b2e81a46f69864e02ec4d42672/users/73a3d3c48913474ab8fa8fedd1bc7d03/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:30.172282 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:52:30.183741 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 798/1580] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:30 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:30.184568 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c67559f-3afb-48cb-bade-32870b612563 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:30.191561 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a29a570-ad7c-40af-aace-fb35ce937c1c admin admin] RBAC: Authorizing `identity:create_grant(project_id=fd9221b2e81a46f69864e02ec4d42672, user_id=73a3d3c48913474ab8fa8fedd1bc7d03, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:30.194315 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a29a570-ad7c-40af-aace-fb35ce937c1c admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:30.195070 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c67559f-3afb-48cb-bade-32870b612563 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:30.200255 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d9624d5-a4b2-41f4-8bae-81cded07c768 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:30.200945 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9624d5-a4b2-41f4-8bae-81cded07c768 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:30.201225 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9624d5-a4b2-41f4-8bae-81cded07c768 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:30.201498 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9624d5-a4b2-41f4-8bae-81cded07c768 admin admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:30.202158 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d9624d5-a4b2-41f4-8bae-81cded07c768 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:30.204557 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d9624d5-a4b2-41f4-8bae-81cded07c768 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:30.207884 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 783/1581] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:52:30 2022] PUT /identity/v3/projects/fd9221b2e81a46f69864e02ec4d42672/users/73a3d3c48913474ab8fa8fedd1bc7d03/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:52:30.208682 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a29a570-ad7c-40af-aace-fb35ce937c1c admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:30.226963 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 799/1582] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:52:30 2022] POST /identity/v3/users => generated 297 bytes in 43 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:52:30.227749 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d9624d5-a4b2-41f4-8bae-81cded07c768 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:30.238497 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d9624d5-a4b2-41f4-8bae-81cded07c768 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:30.271963 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a29a570-ad7c-40af-aace-fb35ce937c1c admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:30.276532 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d451e414-9c7c-4736-874c-b88e21f9b5b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:30.277125 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d451e414-9c7c-4736-874c-b88e21f9b5b2 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:30.277398 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d451e414-9c7c-4736-874c-b88e21f9b5b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:30.277668 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d451e414-9c7c-4736-874c-b88e21f9b5b2 admin admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:30.278292 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d451e414-9c7c-4736-874c-b88e21f9b5b2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:30.280243 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d451e414-9c7c-4736-874c-b88e21f9b5b2 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:30.298312 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-981a01a8-b05e-4fa6-b49c-282a702998f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:30.298882 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981a01a8-b05e-4fa6-b49c-282a702998f9 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:30.299158 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981a01a8-b05e-4fa6-b49c-282a702998f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:30.299420 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981a01a8-b05e-4fa6-b49c-282a702998f9 admin admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:30.300043 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-981a01a8-b05e-4fa6-b49c-282a702998f9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:30.301959 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-981a01a8-b05e-4fa6-b49c-282a702998f9 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:30.303990 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 784/1583] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:52:30 2022] POST /identity/v3/users => generated 282 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:52:30.304962 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d451e414-9c7c-4736-874c-b88e21f9b5b2 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:30.317003 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d451e414-9c7c-4736-874c-b88e21f9b5b2 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:30.322818 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11d6d684-16da-46e2-aa9d-68582f4c20dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:30.323402 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d6d684-16da-46e2-aa9d-68582f4c20dc admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:30.323674 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d6d684-16da-46e2-aa9d-68582f4c20dc admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:30.323966 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d6d684-16da-46e2-aa9d-68582f4c20dc admin admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:30.324589 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11d6d684-16da-46e2-aa9d-68582f4c20dc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:30.326479 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11d6d684-16da-46e2-aa9d-68582f4c20dc admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:30.328632 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 800/1584] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:52:30 2022] POST /identity/v3/users => generated 296 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:52:30.329513 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-981a01a8-b05e-4fa6-b49c-282a702998f9 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:30.339522 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-981a01a8-b05e-4fa6-b49c-282a702998f9 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:30.344781 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf3efb2-bb05-4ffe-a380-456e330ee06c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:30.345376 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf3efb2-bb05-4ffe-a380-456e330ee06c admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:30.345650 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf3efb2-bb05-4ffe-a380-456e330ee06c admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:30.345927 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf3efb2-bb05-4ffe-a380-456e330ee06c admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:30.346565 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cf3efb2-bb05-4ffe-a380-456e330ee06c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:30.348193 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cf3efb2-bb05-4ffe-a380-456e330ee06c admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:30.349300 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 785/1585] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:52:30 2022] POST /identity/v3/users => generated 292 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:52:30.350172 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11d6d684-16da-46e2-aa9d-68582f4c20dc admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:30.361303 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11d6d684-16da-46e2-aa9d-68582f4c20dc admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:30.363228 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 801/1586] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:52:30 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:52:30.364052 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf3efb2-bb05-4ffe-a380-456e330ee06c admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:30.368728 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-185bdd42-171c-46fa-b45c-7976c1dfedea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:30.369329 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185bdd42-171c-46fa-b45c-7976c1dfedea admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:30.369591 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185bdd42-171c-46fa-b45c-7976c1dfedea admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:30.369859 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185bdd42-171c-46fa-b45c-7976c1dfedea admin admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:30.370498 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-185bdd42-171c-46fa-b45c-7976c1dfedea admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:30.372442 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-185bdd42-171c-46fa-b45c-7976c1dfedea admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:30.397014 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 786/1587] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:52:30 2022] POST /identity/v3/users => generated 287 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:52:30.398921 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba3d6001-fb3a-4194-8a4f-181cb3450037 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:30.399405 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba3d6001-fb3a-4194-8a4f-181cb3450037 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:30.399580 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba3d6001-fb3a-4194-8a4f-181cb3450037 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:30.420316 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-ba3d6001-fb3a-4194-8a4f-181cb3450037 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:30.421729 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-ba3d6001-fb3a-4194-8a4f-181cb3450037 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:30.425747 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf3efb2-bb05-4ffe-a380-456e330ee06c admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:30.429380 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 787/1588] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:30.429768 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f0aa7a-baec-4104-9e51-4246c6a9b5b1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:30.430167 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba3d6001-fb3a-4194-8a4f-181cb3450037 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:30.430367 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f0aa7a-baec-4104-9e51-4246c6a9b5b1 service cinder] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:30.430650 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f0aa7a-baec-4104-9e51-4246c6a9b5b1 service cinder] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:30.430942 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f0aa7a-baec-4104-9e51-4246c6a9b5b1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:30.440826 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba3d6001-fb3a-4194-8a4f-181cb3450037 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:30.446012 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c10f1f6-72f0-4575-b6e8-f961684b86fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:30.446615 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c10f1f6-72f0-4575-b6e8-f961684b86fe admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:30.446927 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c10f1f6-72f0-4575-b6e8-f961684b86fe admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:30.447198 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c10f1f6-72f0-4575-b6e8-f961684b86fe admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:30.447845 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c10f1f6-72f0-4575-b6e8-f961684b86fe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:30.449432 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c10f1f6-72f0-4575-b6e8-f961684b86fe admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:30.460043 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 788/1589] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:52:30 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:52:30.460905 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c10f1f6-72f0-4575-b6e8-f961684b86fe admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:30.471064 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c10f1f6-72f0-4575-b6e8-f961684b86fe admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:30.475868 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c555c71-b6c0-482e-8a76-38c68fa07f6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:30.476449 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c555c71-b6c0-482e-8a76-38c68fa07f6d admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:30.476726 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c555c71-b6c0-482e-8a76-38c68fa07f6d admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:30.476995 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c555c71-b6c0-482e-8a76-38c68fa07f6d admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:30.477629 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c555c71-b6c0-482e-8a76-38c68fa07f6d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:30.479239 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c555c71-b6c0-482e-8a76-38c68fa07f6d admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:30.489816 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 789/1590] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:52:30 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:52:30.490688 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c555c71-b6c0-482e-8a76-38c68fa07f6d admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:30.500874 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c555c71-b6c0-482e-8a76-38c68fa07f6d admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:30.505528 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bff3fb6-8455-47be-8548-0ecd4d6cfd27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:30.505695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37f0aa7a-baec-4104-9e51-4246c6a9b5b1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:30.506140 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bff3fb6-8455-47be-8548-0ecd4d6cfd27 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:30.506419 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bff3fb6-8455-47be-8548-0ecd4d6cfd27 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:30.506687 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bff3fb6-8455-47be-8548-0ecd4d6cfd27 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:30.507359 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bff3fb6-8455-47be-8548-0ecd4d6cfd27 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:30.507533 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37f0aa7a-baec-4104-9e51-4246c6a9b5b1 service cinder] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:30.508949 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bff3fb6-8455-47be-8548-0ecd4d6cfd27 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:30.518332 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 802/1591] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:52:30 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 154 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:52:30.519122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f0aa7a-baec-4104-9e51-4246c6a9b5b1 service cinder] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:30.520381 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 790/1592] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:52:30 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:52:30.521305 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bff3fb6-8455-47be-8548-0ecd4d6cfd27 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:30.527978 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:52:30.532965 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bff3fb6-8455-47be-8548-0ecd4d6cfd27 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:30.539970 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f0aa7a-baec-4104-9e51-4246c6a9b5b1 service cinder] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:30.544114 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec8d8fe-eecc-4649-b960-2c726f3c18b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:30.544716 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec8d8fe-eecc-4649-b960-2c726f3c18b1 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:30.544997 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec8d8fe-eecc-4649-b960-2c726f3c18b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:30.545266 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec8d8fe-eecc-4649-b960-2c726f3c18b1 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:30.545938 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cec8d8fe-eecc-4649-b960-2c726f3c18b1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:30.548537 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cec8d8fe-eecc-4649-b960-2c726f3c18b1 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:30.555600 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e128bb3f-c976-44d8-a894-8a05b23ed88f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:30.556268 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e128bb3f-c976-44d8-a894-8a05b23ed88f admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:30.556551 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e128bb3f-c976-44d8-a894-8a05b23ed88f admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:30.557229 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e128bb3f-c976-44d8-a894-8a05b23ed88f admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:30.558067 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e128bb3f-c976-44d8-a894-8a05b23ed88f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:30.559745 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e128bb3f-c976-44d8-a894-8a05b23ed88f admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:30.564331 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 791/1593] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:52:30 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:52:30.565373 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec8d8fe-eecc-4649-b960-2c726f3c18b1 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:30.574905 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 803/1594] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:52:30 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:52:30.574905 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e128bb3f-c976-44d8-a894-8a05b23ed88f admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:30.584544 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec8d8fe-eecc-4649-b960-2c726f3c18b1 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:30.590146 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8557db61-a5eb-4f7a-add2-30ce4962a0f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:30.590753 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8557db61-a5eb-4f7a-add2-30ce4962a0f4 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:30.591068 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8557db61-a5eb-4f7a-add2-30ce4962a0f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:30.591345 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8557db61-a5eb-4f7a-add2-30ce4962a0f4 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:30.591983 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8557db61-a5eb-4f7a-add2-30ce4962a0f4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:30.594296 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8557db61-a5eb-4f7a-add2-30ce4962a0f4 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:30.605045 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 792/1595] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:52:30 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:52:30.605692 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8557db61-a5eb-4f7a-add2-30ce4962a0f4 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:30.617117 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8557db61-a5eb-4f7a-add2-30ce4962a0f4 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:30.621960 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a9b9f7-3316-4b76-88b7-523cd718186b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:30.622557 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a9b9f7-3316-4b76-88b7-523cd718186b admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:30.622865 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a9b9f7-3316-4b76-88b7-523cd718186b admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:30.623142 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a9b9f7-3316-4b76-88b7-523cd718186b admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:30.623936 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92a9b9f7-3316-4b76-88b7-523cd718186b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:30.625786 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92a9b9f7-3316-4b76-88b7-523cd718186b admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:30.637596 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 793/1596] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:52:30 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:52:30.638415 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a9b9f7-3316-4b76-88b7-523cd718186b admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:30.639332 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e128bb3f-c976-44d8-a894-8a05b23ed88f admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:30.644247 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8406fd15-eec6-4c8e-847f-c6cb960c6324 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:30.644868 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8406fd15-eec6-4c8e-847f-c6cb960c6324 service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:30.645248 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8406fd15-eec6-4c8e-847f-c6cb960c6324 service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:30.645526 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8406fd15-eec6-4c8e-847f-c6cb960c6324 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:30.650919 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a9b9f7-3316-4b76-88b7-523cd718186b admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:30.657292 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8406fd15-eec6-4c8e-847f-c6cb960c6324 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:30.657452 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d631a58-feb5-427b-87dd-1a6906716129 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:30.658163 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d631a58-feb5-427b-87dd-1a6906716129 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:30.658536 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d631a58-feb5-427b-87dd-1a6906716129 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:30.658956 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d631a58-feb5-427b-87dd-1a6906716129 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:30.659713 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8406fd15-eec6-4c8e-847f-c6cb960c6324 service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:30.660055 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d631a58-feb5-427b-87dd-1a6906716129 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:30.662003 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d631a58-feb5-427b-87dd-1a6906716129 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:30.676003 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 804/1597] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:52:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:52:30.676919 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8406fd15-eec6-4c8e-847f-c6cb960c6324 service neutron] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:30.684060 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 794/1598] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:52:30 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:52:30.684952 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d631a58-feb5-427b-87dd-1a6906716129 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:30.690093 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8406fd15-eec6-4c8e-847f-c6cb960c6324 service neutron] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:30.704647 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a88117-63a3-426b-a4f4-f1ed032be90f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:30.705269 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a88117-63a3-426b-a4f4-f1ed032be90f admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:30.705533 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a88117-63a3-426b-a4f4-f1ed032be90f admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:30.705844 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a88117-63a3-426b-a4f4-f1ed032be90f admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:30.706430 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d631a58-feb5-427b-87dd-1a6906716129 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:30.706614 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69a88117-63a3-426b-a4f4-f1ed032be90f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:30.708872 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69a88117-63a3-426b-a4f4-f1ed032be90f admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:30.711800 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78ef50a1-7c25-467c-97f9-6abad3a749d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:30.712466 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ef50a1-7c25-467c-97f9-6abad3a749d3 admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:30.712751 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ef50a1-7c25-467c-97f9-6abad3a749d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:30.713031 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ef50a1-7c25-467c-97f9-6abad3a749d3 admin admin] PATH_INFO: `/v3/projects/832fdbe87ca847f094c6cc8a1fc43a59/users/8b1fc2c958f045538fb15d7d1d4a408a/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:30.737898 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 805/1599] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:52:30 2022] GET /identity/v3/roles => generated 1420 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:52:30.737898 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a88117-63a3-426b-a4f4-f1ed032be90f admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:30.739987 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=90,keystone:INSERT=10 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:52:30.751839 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78ef50a1-7c25-467c-97f9-6abad3a749d3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=832fdbe87ca847f094c6cc8a1fc43a59, user_id=8b1fc2c958f045538fb15d7d1d4a408a, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:30.755133 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78ef50a1-7c25-467c-97f9-6abad3a749d3 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:30.767682 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a88117-63a3-426b-a4f4-f1ed032be90f admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:30.774563 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 795/1600] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:52:30 2022] PUT /identity/v3/projects/832fdbe87ca847f094c6cc8a1fc43a59/users/8b1fc2c958f045538fb15d7d1d4a408a/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 90 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:52:30.775850 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78ef50a1-7c25-467c-97f9-6abad3a749d3 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:30.909187 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de44237-ad0f-4ca3-bd9e-0b95623fb74f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:30.910007 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de44237-ad0f-4ca3-bd9e-0b95623fb74f admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:30.910173 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de44237-ad0f-4ca3-bd9e-0b95623fb74f admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:30.910917 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de44237-ad0f-4ca3-bd9e-0b95623fb74f admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:30.911714 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9de44237-ad0f-4ca3-bd9e-0b95623fb74f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:30.913689 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9de44237-ad0f-4ca3-bd9e-0b95623fb74f admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:30.921999 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78ef50a1-7c25-467c-97f9-6abad3a749d3 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:30.925269 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 806/1601] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:52:30 2022] GET /identity/v3/roles => generated 1420 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:52:30.926183 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de44237-ad0f-4ca3-bd9e-0b95623fb74f admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:30.937852 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de44237-ad0f-4ca3-bd9e-0b95623fb74f admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:30.939551 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c014ece-cd81-46fe-8297-1131f1b2c69f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:30.940304 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c014ece-cd81-46fe-8297-1131f1b2c69f admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:30.940745 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c014ece-cd81-46fe-8297-1131f1b2c69f admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:30.941071 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c014ece-cd81-46fe-8297-1131f1b2c69f admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:30.941784 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c014ece-cd81-46fe-8297-1131f1b2c69f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:30.943890 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c014ece-cd81-46fe-8297-1131f1b2c69f admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:30.950119 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1311f18-32d9-4122-bd7a-b95108fcd1e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:30.952122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1311f18-32d9-4122-bd7a-b95108fcd1e0 admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:30.952450 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1311f18-32d9-4122-bd7a-b95108fcd1e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:30.954152 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1311f18-32d9-4122-bd7a-b95108fcd1e0 admin admin] PATH_INFO: `/v3/projects/6a58aefc25e141ca93821a80859875e8/users/5df6979effd84ece8d75b631ba7da196/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:30.963053 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 796/1602] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:52:30 2022] GET /identity/v3/roles => generated 1420 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:52:30.966330 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c014ece-cd81-46fe-8297-1131f1b2c69f admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:30.979381 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c014ece-cd81-46fe-8297-1131f1b2c69f admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:30.985140 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2550231-c82a-4aa8-b730-ddb47870c109 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:30.986161 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2550231-c82a-4aa8-b730-ddb47870c109 admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:30.986545 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2550231-c82a-4aa8-b730-ddb47870c109 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:30.987127 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2550231-c82a-4aa8-b730-ddb47870c109 admin admin] PATH_INFO: `/v3/projects/74b82ca3c7424f5197fe35282cbad2e6/users/432ccb22f1384dc9af452d8eacf7bb14/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:30.995788 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1311f18-32d9-4122-bd7a-b95108fcd1e0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6a58aefc25e141ca93821a80859875e8, user_id=5df6979effd84ece8d75b631ba7da196, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:30.998599 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1311f18-32d9-4122-bd7a-b95108fcd1e0 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:31.011510 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2550231-c82a-4aa8-b730-ddb47870c109 admin admin] RBAC: Authorizing `identity:create_grant(project_id=74b82ca3c7424f5197fe35282cbad2e6, user_id=432ccb22f1384dc9af452d8eacf7bb14, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:31.016269 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2550231-c82a-4aa8-b730-ddb47870c109 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:31.017816 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 807/1603] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:52:30 2022] PUT /identity/v3/projects/6a58aefc25e141ca93821a80859875e8/users/5df6979effd84ece8d75b631ba7da196/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 92 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:52:31.018604 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1311f18-32d9-4122-bd7a-b95108fcd1e0 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:31.046916 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 797/1604] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:52:30 2022] PUT /identity/v3/projects/74b82ca3c7424f5197fe35282cbad2e6/users/432ccb22f1384dc9af452d8eacf7bb14/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 72 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:52:31.046916 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3665edb6-42e6-4926-b90d-138f59ee22d1 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:31.046916 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3665edb6-42e6-4926-b90d-138f59ee22d1 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:31.046916 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3665edb6-42e6-4926-b90d-138f59ee22d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:31.063568 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-3665edb6-42e6-4926-b90d-138f59ee22d1 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:31.064847 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-3665edb6-42e6-4926-b90d-138f59ee22d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:31.134856 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:52:31.145306 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1311f18-32d9-4122-bd7a-b95108fcd1e0 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:31.154262 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbca9010-4565-4ef6-aea2-5d7d83fa4ac5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:31.155802 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbca9010-4565-4ef6-aea2-5d7d83fa4ac5 admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:31.156296 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbca9010-4565-4ef6-aea2-5d7d83fa4ac5 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:31.157133 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbca9010-4565-4ef6-aea2-5d7d83fa4ac5 admin admin] PATH_INFO: `/v3/projects/43c4afbc769248eb8c4f90c337a3d28b/users/84cbd86bbf99414090fc704b9d0817bf/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:31.157666 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 798/1605] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:31.158180 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3665edb6-42e6-4926-b90d-138f59ee22d1 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:31.170030 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3665edb6-42e6-4926-b90d-138f59ee22d1 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:31.176361 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aedb19ed-40ee-4423-9dcf-c50b38dd86c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:31.177021 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aedb19ed-40ee-4423-9dcf-c50b38dd86c6 admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:31.177303 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aedb19ed-40ee-4423-9dcf-c50b38dd86c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:31.177567 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aedb19ed-40ee-4423-9dcf-c50b38dd86c6 admin admin] PATH_INFO: `/v3/projects/172e557567404e0f906fd1c3452148d8/users/d886b168a8eb4424987555c073d71727/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:31.180630 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbca9010-4565-4ef6-aea2-5d7d83fa4ac5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=43c4afbc769248eb8c4f90c337a3d28b, user_id=84cbd86bbf99414090fc704b9d0817bf, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:31.184761 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbca9010-4565-4ef6-aea2-5d7d83fa4ac5 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:31.196729 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aedb19ed-40ee-4423-9dcf-c50b38dd86c6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=172e557567404e0f906fd1c3452148d8, user_id=d886b168a8eb4424987555c073d71727, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:31.199483 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aedb19ed-40ee-4423-9dcf-c50b38dd86c6 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:31.218226 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 808/1606] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:52:31 2022] PUT /identity/v3/projects/43c4afbc769248eb8c4f90c337a3d28b/users/84cbd86bbf99414090fc704b9d0817bf/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:52:31.223181 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbca9010-4565-4ef6-aea2-5d7d83fa4ac5 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:31.229066 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 799/1607] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:52:31 2022] PUT /identity/v3/projects/172e557567404e0f906fd1c3452148d8/users/d886b168a8eb4424987555c073d71727/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 71 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:52:31.232864 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9078654-83af-4200-a5fb-328077dce41e None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:31.232864 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9078654-83af-4200-a5fb-328077dce41e None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:31.232864 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9078654-83af-4200-a5fb-328077dce41e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:31.274131 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-f9078654-83af-4200-a5fb-328077dce41e None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:31.275590 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-f9078654-83af-4200-a5fb-328077dce41e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:31.321363 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:52:31.332610 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbca9010-4565-4ef6-aea2-5d7d83fa4ac5 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:31.399414 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 800/1608] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:31.400668 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f4f451-6e4a-480e-9ef6-091ba99a7f4c None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:31.400960 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f4f451-6e4a-480e-9ef6-091ba99a7f4c None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:31.401297 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f4f451-6e4a-480e-9ef6-091ba99a7f4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:31.427113 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5801582-5f70-445b-9c98-16d86592bb44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:31.428142 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5801582-5f70-445b-9c98-16d86592bb44 admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:31.428519 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5801582-5f70-445b-9c98-16d86592bb44 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:31.428889 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5801582-5f70-445b-9c98-16d86592bb44 admin admin] PATH_INFO: `/v3/projects/f4d1bc2ff0c7459ea7ef2aba844623ca/users/3902a36ec1354608a9d1d7789091a8cf/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:31.450903 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-a7f4f451-6e4a-480e-9ef6-091ba99a7f4c None None] MFA Rules not processed for user `432ccb22f1384dc9af452d8eacf7bb14`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:31.452274 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5801582-5f70-445b-9c98-16d86592bb44 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f4d1bc2ff0c7459ea7ef2aba844623ca, user_id=3902a36ec1354608a9d1d7789091a8cf, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:31.453040 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-a7f4f451-6e4a-480e-9ef6-091ba99a7f4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:31.456776 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5801582-5f70-445b-9c98-16d86592bb44 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:31.483839 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 809/1609] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:52:31 2022] PUT /identity/v3/projects/f4d1bc2ff0c7459ea7ef2aba844623ca/users/3902a36ec1354608a9d1d7789091a8cf/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 254 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:52:31.484902 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5801582-5f70-445b-9c98-16d86592bb44 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:31.486903 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-f5801582-5f70-445b-9c98-16d86592bb44 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:31.537519 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 801/1610] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:31 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:31.539253 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e14936b5-a74f-41ab-93b1-87fb36beeda3 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:31.539537 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e14936b5-a74f-41ab-93b1-87fb36beeda3 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:31.539811 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e14936b5-a74f-41ab-93b1-87fb36beeda3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:31.550710 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5801582-5f70-445b-9c98-16d86592bb44 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:31.556579 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9705a8e0-bc14-4723-9e1b-17eec69097a0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:31.556837 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9705a8e0-bc14-4723-9e1b-17eec69097a0 service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:31.557232 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9705a8e0-bc14-4723-9e1b-17eec69097a0 service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:31.557619 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9705a8e0-bc14-4723-9e1b-17eec69097a0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:31.560689 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-e14936b5-a74f-41ab-93b1-87fb36beeda3 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:31.562601 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-e14936b5-a74f-41ab-93b1-87fb36beeda3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:31.583898 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:52:31.630182 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9705a8e0-bc14-4723-9e1b-17eec69097a0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:31.631677 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9705a8e0-bc14-4723-9e1b-17eec69097a0 service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:31.635785 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 802/1611] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:31.636847 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35452dfe-443a-4863-88d0-e683359eb053 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:31.637169 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35452dfe-443a-4863-88d0-e683359eb053 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:31.637448 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35452dfe-443a-4863-88d0-e683359eb053 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:31.643629 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 810/1612] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:52:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 159 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:52:31.644538 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9705a8e0-bc14-4723-9e1b-17eec69097a0 service neutron] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:31.652721 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9705a8e0-bc14-4723-9e1b-17eec69097a0 service neutron] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:31.657504 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa7b2bb-30cf-425c-8ca0-150ce42b1c3c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:31.658834 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa7b2bb-30cf-425c-8ca0-150ce42b1c3c service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:31.659125 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa7b2bb-30cf-425c-8ca0-150ce42b1c3c service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:31.659528 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa7b2bb-30cf-425c-8ca0-150ce42b1c3c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:31.672927 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aa7b2bb-30cf-425c-8ca0-150ce42b1c3c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:31.674883 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aa7b2bb-30cf-425c-8ca0-150ce42b1c3c service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:31.675033 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-35452dfe-443a-4863-88d0-e683359eb053 None None] MFA Rules not processed for user `84cbd86bbf99414090fc704b9d0817bf`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:31.676348 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-35452dfe-443a-4863-88d0-e683359eb053 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:31.686636 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 811/1613] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:52:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:52:31.688315 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77012b05-48da-4cd0-8451-a89f874fc250 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:31.688748 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77012b05-48da-4cd0-8451-a89f874fc250 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:31.689187 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77012b05-48da-4cd0-8451-a89f874fc250 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:31.727927 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-77012b05-48da-4cd0-8451-a89f874fc250 None None] MFA Rules not processed for user `3902a36ec1354608a9d1d7789091a8cf`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:31.729742 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-77012b05-48da-4cd0-8451-a89f874fc250 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:31.765112 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:52:31.773448 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 803/1614] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:31 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:31.774538 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35452dfe-443a-4863-88d0-e683359eb053 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:31.793940 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35452dfe-443a-4863-88d0-e683359eb053 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:31.800897 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9d908b-1cbc-4399-9d4a-7dfee5d3c031 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:31.802017 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9d908b-1cbc-4399-9d4a-7dfee5d3c031 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:31.802301 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9d908b-1cbc-4399-9d4a-7dfee5d3c031 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:31.803126 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9d908b-1cbc-4399-9d4a-7dfee5d3c031 admin admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:31.804343 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c9d908b-1cbc-4399-9d4a-7dfee5d3c031 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:31.806899 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c9d908b-1cbc-4399-9d4a-7dfee5d3c031 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:31.834288 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 812/1615] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:31 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:31.834288 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77012b05-48da-4cd0-8451-a89f874fc250 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:31.849891 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77012b05-48da-4cd0-8451-a89f874fc250 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:31.855656 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 804/1616] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:52:31 2022] POST /identity/v3/projects => generated 363 bytes in 81 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:52:31.856567 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9d908b-1cbc-4399-9d4a-7dfee5d3c031 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:31.860445 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41a0b04e-81e1-4cfe-9dae-94afd04f6a6c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:31.861136 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a0b04e-81e1-4cfe-9dae-94afd04f6a6c service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:31.861321 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a0b04e-81e1-4cfe-9dae-94afd04f6a6c service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:31.861597 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a0b04e-81e1-4cfe-9dae-94afd04f6a6c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:31.871789 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41a0b04e-81e1-4cfe-9dae-94afd04f6a6c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:31.873249 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41a0b04e-81e1-4cfe-9dae-94afd04f6a6c service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:31.884642 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 813/1617] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:52:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:52:31.885505 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41a0b04e-81e1-4cfe-9dae-94afd04f6a6c service neutron] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:31.908601 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41a0b04e-81e1-4cfe-9dae-94afd04f6a6c service neutron] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:32.000556 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e3e9a0-2712-4c33-938a-2b8ad2f67327 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:32.002058 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e3e9a0-2712-4c33-938a-2b8ad2f67327 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:32.002374 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e3e9a0-2712-4c33-938a-2b8ad2f67327 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:32.002669 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e3e9a0-2712-4c33-938a-2b8ad2f67327 admin admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:32.004565 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e3e9a0-2712-4c33-938a-2b8ad2f67327 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:32.006929 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e3e9a0-2712-4c33-938a-2b8ad2f67327 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:32.008527 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9d908b-1cbc-4399-9d4a-7dfee5d3c031 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:32.020574 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ecf080d-e5a1-4898-b3a1-223bc134b6e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:32.021371 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ecf080d-e5a1-4898-b3a1-223bc134b6e1 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:32.021819 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ecf080d-e5a1-4898-b3a1-223bc134b6e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:32.022136 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ecf080d-e5a1-4898-b3a1-223bc134b6e1 admin admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:32.022983 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ecf080d-e5a1-4898-b3a1-223bc134b6e1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:32.025084 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ecf080d-e5a1-4898-b3a1-223bc134b6e1 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:32.077475 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 814/1618] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:52:31 2022] POST /identity/v3/projects => generated 373 bytes in 192 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:52:32.079804 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e3e9a0-2712-4c33-938a-2b8ad2f67327 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:32.088855 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 805/1619] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:52:31 2022] POST /identity/v3/projects => generated 393 bytes in 232 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:52:32.091652 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:52:32.093085 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ecf080d-e5a1-4898-b3a1-223bc134b6e1 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:32.195657 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e3e9a0-2712-4c33-938a-2b8ad2f67327 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:32.244543 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ecf080d-e5a1-4898-b3a1-223bc134b6e1 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:32.251710 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3213bf2a-28d5-49b2-93cd-98f4c9c0f0bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:32.252608 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3213bf2a-28d5-49b2-93cd-98f4c9c0f0bd admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:32.252979 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3213bf2a-28d5-49b2-93cd-98f4c9c0f0bd admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:32.253344 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3213bf2a-28d5-49b2-93cd-98f4c9c0f0bd admin admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:32.254195 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3213bf2a-28d5-49b2-93cd-98f4c9c0f0bd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:32.256257 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3213bf2a-28d5-49b2-93cd-98f4c9c0f0bd admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:32.284353 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 806/1620] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:52:32 2022] POST /identity/v3/users => generated 293 bytes in 192 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:52:32.285778 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3213bf2a-28d5-49b2-93cd-98f4c9c0f0bd admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:32.302132 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3213bf2a-28d5-49b2-93cd-98f4c9c0f0bd admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:32.307774 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83dd8847-3881-45f5-a748-d03ea2a2dd4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:32.308518 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83dd8847-3881-45f5-a748-d03ea2a2dd4e admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:32.308911 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83dd8847-3881-45f5-a748-d03ea2a2dd4e admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:32.309268 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83dd8847-3881-45f5-a748-d03ea2a2dd4e admin admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:32.310186 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83dd8847-3881-45f5-a748-d03ea2a2dd4e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:32.312644 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83dd8847-3881-45f5-a748-d03ea2a2dd4e admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:32.315135 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb0650a-5918-482a-8b1e-95c54da91d28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:32.315765 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb0650a-5918-482a-8b1e-95c54da91d28 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:32.316047 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb0650a-5918-482a-8b1e-95c54da91d28 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:32.316323 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb0650a-5918-482a-8b1e-95c54da91d28 admin admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:32.316970 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcb0650a-5918-482a-8b1e-95c54da91d28 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:32.318919 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcb0650a-5918-482a-8b1e-95c54da91d28 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:32.339773 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 807/1621] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:52:32 2022] POST /identity/v3/users => generated 303 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:52:32.346715 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83dd8847-3881-45f5-a748-d03ea2a2dd4e admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:32.365280 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83dd8847-3881-45f5-a748-d03ea2a2dd4e admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:32.371869 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 815/1622] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:52:32 2022] POST /identity/v3/users => generated 288 bytes in 294 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:52:32.372719 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb0650a-5918-482a-8b1e-95c54da91d28 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:32.378777 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-625c4a71-434b-4f8b-a7ba-33e3a06052c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:32.379451 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625c4a71-434b-4f8b-a7ba-33e3a06052c7 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:32.379736 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625c4a71-434b-4f8b-a7ba-33e3a06052c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:32.380021 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625c4a71-434b-4f8b-a7ba-33e3a06052c7 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:32.380720 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-625c4a71-434b-4f8b-a7ba-33e3a06052c7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:32.382349 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-625c4a71-434b-4f8b-a7ba-33e3a06052c7 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:32.397813 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb0650a-5918-482a-8b1e-95c54da91d28 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:32.399608 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 808/1623] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:52:32 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:52:32.400544 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-625c4a71-434b-4f8b-a7ba-33e3a06052c7 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:32.409180 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7f3138-a638-4871-992b-0ad2083e5682 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:32.412666 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-625c4a71-434b-4f8b-a7ba-33e3a06052c7 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:32.413230 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7f3138-a638-4871-992b-0ad2083e5682 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:32.413666 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7f3138-a638-4871-992b-0ad2083e5682 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:32.414087 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7f3138-a638-4871-992b-0ad2083e5682 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:32.414937 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f7f3138-a638-4871-992b-0ad2083e5682 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:32.416716 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f7f3138-a638-4871-992b-0ad2083e5682 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:32.422416 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00680e1f-7b7d-4b89-96a5-c09926d328c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:32.423079 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00680e1f-7b7d-4b89-96a5-c09926d328c3 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:32.423365 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00680e1f-7b7d-4b89-96a5-c09926d328c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:32.423636 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00680e1f-7b7d-4b89-96a5-c09926d328c3 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:32.424750 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00680e1f-7b7d-4b89-96a5-c09926d328c3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:32.426155 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00680e1f-7b7d-4b89-96a5-c09926d328c3 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:32.439698 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 809/1624] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:52:32 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:52:32.440642 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00680e1f-7b7d-4b89-96a5-c09926d328c3 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:32.453622 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00680e1f-7b7d-4b89-96a5-c09926d328c3 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:32.459514 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45322976-42bb-41a6-a61e-b765f8e8d71c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:32.460250 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45322976-42bb-41a6-a61e-b765f8e8d71c admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:32.460538 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45322976-42bb-41a6-a61e-b765f8e8d71c admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:32.460815 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45322976-42bb-41a6-a61e-b765f8e8d71c admin admin] PATH_INFO: `/v3/projects/47ed436614e34447b8610dee2ffe06cc/users/788e95ddb109462e81055cbd0653b99d/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:32.470834 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:52:32.471982 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 816/1625] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:52:32 2022] GET /identity/v3/roles => generated 1420 bytes in 100 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:52:32.472896 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7f3138-a638-4871-992b-0ad2083e5682 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:32.485589 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45322976-42bb-41a6-a61e-b765f8e8d71c admin admin] RBAC: Authorizing `identity:create_grant(project_id=47ed436614e34447b8610dee2ffe06cc, user_id=788e95ddb109462e81055cbd0653b99d, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:32.489580 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45322976-42bb-41a6-a61e-b765f8e8d71c admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:32.507817 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7f3138-a638-4871-992b-0ad2083e5682 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:32.528232 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 810/1626] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:52:32 2022] PUT /identity/v3/projects/47ed436614e34447b8610dee2ffe06cc/users/788e95ddb109462e81055cbd0653b99d/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 85 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:52:32.529370 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45322976-42bb-41a6-a61e-b765f8e8d71c admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:32.639603 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45322976-42bb-41a6-a61e-b765f8e8d71c admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:32.647386 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6953d4-5d39-4194-a6f6-ea679496071e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:32.648276 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6953d4-5d39-4194-a6f6-ea679496071e admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:32.648566 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6953d4-5d39-4194-a6f6-ea679496071e admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:32.648850 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6953d4-5d39-4194-a6f6-ea679496071e admin admin] PATH_INFO: `/v3/projects/d93d1761f7564b53883e3724fe85cc68/users/27219727f89444e38cb27313860fbd11/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:32.670830 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc6953d4-5d39-4194-a6f6-ea679496071e admin admin] RBAC: Authorizing `identity:create_grant(project_id=d93d1761f7564b53883e3724fe85cc68, user_id=27219727f89444e38cb27313860fbd11, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:32.673656 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbedae6f-81b9-495d-a95a-1a5501093dc4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:32.674140 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc6953d4-5d39-4194-a6f6-ea679496071e admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:32.674331 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbedae6f-81b9-495d-a95a-1a5501093dc4 admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:32.674620 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbedae6f-81b9-495d-a95a-1a5501093dc4 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:32.674948 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbedae6f-81b9-495d-a95a-1a5501093dc4 admin admin] PATH_INFO: `/v3/projects/5e8b0cb57b5c40dfbefb897953d81c7e/users/5780ba3d9ec3462dab4bc731886be859/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:32.707555 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 811/1627] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:52:32 2022] PUT /identity/v3/projects/d93d1761f7564b53883e3724fe85cc68/users/27219727f89444e38cb27313860fbd11/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:52:32.713408 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbedae6f-81b9-495d-a95a-1a5501093dc4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5e8b0cb57b5c40dfbefb897953d81c7e, user_id=5780ba3d9ec3462dab4bc731886be859, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:32.716105 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6953d4-5d39-4194-a6f6-ea679496071e admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:32.716266 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbedae6f-81b9-495d-a95a-1a5501093dc4 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:32.745242 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 817/1628] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:52:32 2022] PUT /identity/v3/projects/5e8b0cb57b5c40dfbefb897953d81c7e/users/5780ba3d9ec3462dab4bc731886be859/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 272 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:52:32.746157 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbedae6f-81b9-495d-a95a-1a5501093dc4 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:32.788579 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:52:32.884108 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbedae6f-81b9-495d-a95a-1a5501093dc4 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:32.901841 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a35df5-2ad1-4bef-8ee1-f9e408185236 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:32.903641 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a35df5-2ad1-4bef-8ee1-f9e408185236 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:32.904848 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a35df5-2ad1-4bef-8ee1-f9e408185236 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:32.905983 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a35df5-2ad1-4bef-8ee1-f9e408185236 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:32.909825 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a35df5-2ad1-4bef-8ee1-f9e408185236 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:32.912171 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a35df5-2ad1-4bef-8ee1-f9e408185236 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:32.915726 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6953d4-5d39-4194-a6f6-ea679496071e admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:32.929828 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc3173e3-9b16-417e-b530-d99cac9192dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:32.930647 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3173e3-9b16-417e-b530-d99cac9192dd admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:32.930966 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3173e3-9b16-417e-b530-d99cac9192dd admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:32.931238 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3173e3-9b16-417e-b530-d99cac9192dd admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:32.932147 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc3173e3-9b16-417e-b530-d99cac9192dd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:32.934100 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc3173e3-9b16-417e-b530-d99cac9192dd admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:32.939959 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 818/1629] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:52:32 2022] GET /identity/v3/roles => generated 1420 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:52:32.941789 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a35df5-2ad1-4bef-8ee1-f9e408185236 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:32.957323 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 812/1630] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:52:32 2022] GET /identity/v3/roles => generated 1420 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:52:32.959016 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc3173e3-9b16-417e-b530-d99cac9192dd admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:32.967976 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a35df5-2ad1-4bef-8ee1-f9e408185236 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:32.974272 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc3173e3-9b16-417e-b530-d99cac9192dd admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:32.979512 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7555b9b4-adf7-4845-918c-b99994de318e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:32.980475 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7555b9b4-adf7-4845-918c-b99994de318e admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:32.980732 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7555b9b4-adf7-4845-918c-b99994de318e admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:32.980950 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7555b9b4-adf7-4845-918c-b99994de318e admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:32.981650 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7555b9b4-adf7-4845-918c-b99994de318e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:32.983189 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1357be4b-cdeb-4c97-9a00-5044719b6549 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:32.983346 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7555b9b4-adf7-4845-918c-b99994de318e admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:32.985136 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1357be4b-cdeb-4c97-9a00-5044719b6549 admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:32.986010 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1357be4b-cdeb-4c97-9a00-5044719b6549 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:32.986298 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1357be4b-cdeb-4c97-9a00-5044719b6549 admin admin] PATH_INFO: `/v3/projects/d93d1761f7564b53883e3724fe85cc68/users/27219727f89444e38cb27313860fbd11/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:32.989256 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1357be4b-cdeb-4c97-9a00-5044719b6549 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d93d1761f7564b53883e3724fe85cc68, user_id=27219727f89444e38cb27313860fbd11, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:32.993627 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1357be4b-cdeb-4c97-9a00-5044719b6549 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:33.007672 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 819/1631] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:52:32 2022] GET /identity/v3/roles => generated 1420 bytes in 67 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:52:33.008592 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7555b9b4-adf7-4845-918c-b99994de318e admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:33.018983 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 813/1632] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:52:32 2022] PUT /identity/v3/projects/d93d1761f7564b53883e3724fe85cc68/users/27219727f89444e38cb27313860fbd11/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:52:33.021587 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1357be4b-cdeb-4c97-9a00-5044719b6549 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:33.031426 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7555b9b4-adf7-4845-918c-b99994de318e admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:33.136631 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dde3d6e2-9c93-46a7-9899-573d247a9270 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:33.137748 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde3d6e2-9c93-46a7-9899-573d247a9270 admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:33.138194 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde3d6e2-9c93-46a7-9899-573d247a9270 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:33.139101 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde3d6e2-9c93-46a7-9899-573d247a9270 admin admin] PATH_INFO: `/v3/projects/47ed436614e34447b8610dee2ffe06cc/users/788e95ddb109462e81055cbd0653b99d/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:33.143581 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dde3d6e2-9c93-46a7-9899-573d247a9270 admin admin] RBAC: Authorizing `identity:create_grant(project_id=47ed436614e34447b8610dee2ffe06cc, user_id=788e95ddb109462e81055cbd0653b99d, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:33.146106 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:52:33.148076 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dde3d6e2-9c93-46a7-9899-573d247a9270 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:33.166011 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1357be4b-cdeb-4c97-9a00-5044719b6549 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:33.174990 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 820/1633] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:52:33 2022] PUT /identity/v3/projects/47ed436614e34447b8610dee2ffe06cc/users/788e95ddb109462e81055cbd0653b99d/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:52:33.176308 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee03340a-6451-4dc8-accf-9d5d43415ab9 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:33.176597 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee03340a-6451-4dc8-accf-9d5d43415ab9 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:33.176869 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee03340a-6451-4dc8-accf-9d5d43415ab9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:33.257793 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-ee03340a-6451-4dc8-accf-9d5d43415ab9 None None] MFA Rules not processed for user `27219727f89444e38cb27313860fbd11`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:33.259490 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-ee03340a-6451-4dc8-accf-9d5d43415ab9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:33.283571 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d6da9d-5442-427d-aa46-5c4141dc7bce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:33.284272 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d6da9d-5442-427d-aa46-5c4141dc7bce admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:33.284515 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d6da9d-5442-427d-aa46-5c4141dc7bce admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:33.284787 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d6da9d-5442-427d-aa46-5c4141dc7bce admin admin] PATH_INFO: `/v3/projects/5e8b0cb57b5c40dfbefb897953d81c7e/users/5780ba3d9ec3462dab4bc731886be859/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:33.288078 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d6da9d-5442-427d-aa46-5c4141dc7bce admin admin] RBAC: Authorizing `identity:create_grant(project_id=5e8b0cb57b5c40dfbefb897953d81c7e, user_id=5780ba3d9ec3462dab4bc731886be859, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:33.291039 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d6da9d-5442-427d-aa46-5c4141dc7bce admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:33.325602 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 814/1634] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:52:33 2022] PUT /identity/v3/projects/5e8b0cb57b5c40dfbefb897953d81c7e/users/5780ba3d9ec3462dab4bc731886be859/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 297 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:52:33.325602 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea22705-22c5-4c16-905f-a6f5ed886275 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:33.325602 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea22705-22c5-4c16-905f-a6f5ed886275 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:33.325602 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea22705-22c5-4c16-905f-a6f5ed886275 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:33.471985 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-4ea22705-22c5-4c16-905f-a6f5ed886275 None None] MFA Rules not processed for user `788e95ddb109462e81055cbd0653b99d`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:33.471985 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-4ea22705-22c5-4c16-905f-a6f5ed886275 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:33.479621 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 821/1635] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:33 2022] POST /identity/v3/auth/tokens => generated 3337 bytes in 282 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:33.479621 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21763af-e395-47c8-8bcf-5c379cfeef77 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:33.479621 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21763af-e395-47c8-8bcf-5c379cfeef77 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:33.479621 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21763af-e395-47c8-8bcf-5c379cfeef77 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:33.479621 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 822/1636] 10.222.0.27 () {54 vars in 962 bytes} [Sun Sep 11 00:52:33 2022] GET /identity => generated 270 bytes in 15 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:52:33.489844 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b57f6ad-f878-4e7c-9154-19070340ebf1 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:33.489844 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b57f6ad-f878-4e7c-9154-19070340ebf1 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:33.490359 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b57f6ad-f878-4e7c-9154-19070340ebf1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:33.560947 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-5b57f6ad-f878-4e7c-9154-19070340ebf1 None None] MFA Rules not processed for user `5780ba3d9ec3462dab4bc731886be859`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:33.564400 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-5b57f6ad-f878-4e7c-9154-19070340ebf1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:33.590359 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:52:33.605660 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 815/1637] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:33 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 288 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:33.606579 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ea22705-22c5-4c16-905f-a6f5ed886275 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:33.743072 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ea22705-22c5-4c16-905f-a6f5ed886275 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:33.753409 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fdc9fe5-3d1c-430f-801d-df93c8892234 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:33.754032 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fdc9fe5-3d1c-430f-801d-df93c8892234 service cinder] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:33.754327 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fdc9fe5-3d1c-430f-801d-df93c8892234 service cinder] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:33.754949 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fdc9fe5-3d1c-430f-801d-df93c8892234 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:33.767400 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 823/1638] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:33 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 293 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:33.768757 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6288cf0-5c67-47c1-a407-b64e11ab6ae4 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:33.769050 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6288cf0-5c67-47c1-a407-b64e11ab6ae4 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:33.769343 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6288cf0-5c67-47c1-a407-b64e11ab6ae4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:33.880122 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:52:33.883552 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-d6288cf0-5c67-47c1-a407-b64e11ab6ae4 None None] MFA Rules not processed for user `835def0e6c6d4365a2767051509eb0d6`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:33.886045 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-d6288cf0-5c67-47c1-a407-b64e11ab6ae4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:33.934668 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fdc9fe5-3d1c-430f-801d-df93c8892234 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:33.940122 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fdc9fe5-3d1c-430f-801d-df93c8892234 service cinder] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:33.965660 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 816/1639] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:52:33 2022] GET /identity/v3/auth/tokens => generated 3337 bytes in 359 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:52:33.977822 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fdc9fe5-3d1c-430f-801d-df93c8892234 service cinder] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:33.996709 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fdc9fe5-3d1c-430f-801d-df93c8892234 service cinder] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:34.008009 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5fdf44c-ff8e-48a8-8d6d-9805641ad97d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:34.008631 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5fdf44c-ff8e-48a8-8d6d-9805641ad97d service cinder] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:34.008905 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5fdf44c-ff8e-48a8-8d6d-9805641ad97d service cinder] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:34.009318 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5fdf44c-ff8e-48a8-8d6d-9805641ad97d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:34.026417 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 824/1640] 10.222.0.27 () {58 vars in 1059 bytes} [Sun Sep 11 00:52:33 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 258 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:34.031271 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f66ff53-c210-4d57-9258-856b4265cbf9 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:34.031559 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f66ff53-c210-4d57-9258-856b4265cbf9 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:34.031864 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f66ff53-c210-4d57-9258-856b4265cbf9 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:34.036822 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 825/1641] 10.222.0.27 () {54 vars in 912 bytes} [Sun Sep 11 00:52:34 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:52:34.041026 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5fdf44c-ff8e-48a8-8d6d-9805641ad97d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:34.042550 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc5e005-66f9-4f37-92dc-97992f217c06 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:34.042960 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc5e005-66f9-4f37-92dc-97992f217c06 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:34.043448 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc5e005-66f9-4f37-92dc-97992f217c06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:34.044325 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5fdf44c-ff8e-48a8-8d6d-9805641ad97d service cinder] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:34.086392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 817/1642] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:52:33 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:52:34.105565 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-edc5e005-66f9-4f37-92dc-97992f217c06 None None] MFA Rules not processed for user `835def0e6c6d4365a2767051509eb0d6`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:34.105700 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5fdf44c-ff8e-48a8-8d6d-9805641ad97d service cinder] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:34.107853 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-edc5e005-66f9-4f37-92dc-97992f217c06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:34.117541 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5fdf44c-ff8e-48a8-8d6d-9805641ad97d service cinder] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:34.141467 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-092da709-e90b-4c0d-a05e-d558892079d8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:34.142442 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092da709-e90b-4c0d-a05e-d558892079d8 service cinder] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:34.142751 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092da709-e90b-4c0d-a05e-d558892079d8 service cinder] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:34.143099 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092da709-e90b-4c0d-a05e-d558892079d8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:34.143601 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 826/1643] 10.222.0.27 () {58 vars in 1009 bytes} [Sun Sep 11 00:52:34 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:34.173849 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-092da709-e90b-4c0d-a05e-d558892079d8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:34.176486 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-092da709-e90b-4c0d-a05e-d558892079d8 service cinder] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:34.219878 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edc5e005-66f9-4f37-92dc-97992f217c06 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:34.220279 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 818/1644] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:52:34 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 11 00:52:34.232305 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edc5e005-66f9-4f37-92dc-97992f217c06 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:34.242832 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df10a972-bee5-4032-a9c1-cae617c21841 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:34.243468 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df10a972-bee5-4032-a9c1-cae617c21841 service cinder] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:34.243746 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df10a972-bee5-4032-a9c1-cae617c21841 service cinder] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:34.244017 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df10a972-bee5-4032-a9c1-cae617c21841 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:34.297048 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-092da709-e90b-4c0d-a05e-d558892079d8 service cinder] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:34.312815 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-092da709-e90b-4c0d-a05e-d558892079d8 service cinder] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:34.323495 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d9f6dc0-40bf-4178-9594-ae28d4df23c4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:34.324578 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9f6dc0-40bf-4178-9594-ae28d4df23c4 service cinder] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:34.325079 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9f6dc0-40bf-4178-9594-ae28d4df23c4 service cinder] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:34.325652 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9f6dc0-40bf-4178-9594-ae28d4df23c4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:34.343158 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d9f6dc0-40bf-4178-9594-ae28d4df23c4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:34.351485 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d9f6dc0-40bf-4178-9594-ae28d4df23c4 service cinder] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:34.372746 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df10a972-bee5-4032-a9c1-cae617c21841 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:34.374282 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df10a972-bee5-4032-a9c1-cae617c21841 service cinder] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:34.381930 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 819/1645] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:52:34 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:52:34.404503 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 827/1646] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:52:34 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 185 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:52:36.178301 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d9f6dc0-40bf-4178-9594-ae28d4df23c4 service cinder] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:36.227218 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df10a972-bee5-4032-a9c1-cae617c21841 service cinder] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:36.266293 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:52:36.434240 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d9f6dc0-40bf-4178-9594-ae28d4df23c4 service cinder] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:36.452180 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd66f904-e4d1-4174-91cf-38e102d500c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:36.452894 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd66f904-e4d1-4174-91cf-38e102d500c7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:36.453170 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd66f904-e4d1-4174-91cf-38e102d500c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:36.453470 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd66f904-e4d1-4174-91cf-38e102d500c7 admin admin] PATH_INFO: `/v3/users/3902a36ec1354608a9d1d7789091a8cf` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:36.457966 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd66f904-e4d1-4174-91cf-38e102d500c7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3902a36ec1354608a9d1d7789091a8cf)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:36.458883 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: WARNING py.warnings [None req-bd66f904-e4d1-4174-91cf-38e102d500c7 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 11 00:52:36.458883 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: warnings.warn(msg) Sep 11 00:52:36.458883 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: Sep 11 00:52:36.460102 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd66f904-e4d1-4174-91cf-38e102d500c7 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:36.469937 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df10a972-bee5-4032-a9c1-cae617c21841 service cinder] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:36.488535 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ab079c-857a-4ef1-8110-ae466d826dc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:36.489901 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ab079c-857a-4ef1-8110-ae466d826dc8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:36.490186 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ab079c-857a-4ef1-8110-ae466d826dc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:36.490461 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ab079c-857a-4ef1-8110-ae466d826dc8 admin admin] PATH_INFO: `/v3/users/432ccb22f1384dc9af452d8eacf7bb14` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:36.499344 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62ab079c-857a-4ef1-8110-ae466d826dc8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=432ccb22f1384dc9af452d8eacf7bb14)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:36.504716 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62ab079c-857a-4ef1-8110-ae466d826dc8 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:36.662849 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-62ab079c-857a-4ef1-8110-ae466d826dc8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '432ccb22f1384dc9af452d8eacf7bb14'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:36.664565 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-bd66f904-e4d1-4174-91cf-38e102d500c7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3902a36ec1354608a9d1d7789091a8cf'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:36.664897 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-62ab079c-857a-4ef1-8110-ae466d826dc8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '432ccb22f1384dc9af452d8eacf7bb14'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:36.665322 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-bd66f904-e4d1-4174-91cf-38e102d500c7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3902a36ec1354608a9d1d7789091a8cf'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:36.665767 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-62ab079c-857a-4ef1-8110-ae466d826dc8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '432ccb22f1384dc9af452d8eacf7bb14'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:36.666182 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-bd66f904-e4d1-4174-91cf-38e102d500c7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3902a36ec1354608a9d1d7789091a8cf'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:36.695324 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-62ab079c-857a-4ef1-8110-ae466d826dc8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '432ccb22f1384dc9af452d8eacf7bb14'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:36.709698 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-bd66f904-e4d1-4174-91cf-38e102d500c7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3902a36ec1354608a9d1d7789091a8cf'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:36.727316 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-62ab079c-857a-4ef1-8110-ae466d826dc8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '432ccb22f1384dc9af452d8eacf7bb14'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:36.753406 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 828/1647] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:52:36 2022] DELETE /identity/v3/users/432ccb22f1384dc9af452d8eacf7bb14 => generated 0 bytes in 527 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:52:36.759453 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaff4757-695b-4c57-83ef-0a2aa4bae0e0 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:36.759754 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaff4757-695b-4c57-83ef-0a2aa4bae0e0 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:36.760069 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaff4757-695b-4c57-83ef-0a2aa4bae0e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:36.774560 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-bd66f904-e4d1-4174-91cf-38e102d500c7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3902a36ec1354608a9d1d7789091a8cf'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:36.813349 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 820/1648] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:52:36 2022] DELETE /identity/v3/users/3902a36ec1354608a9d1d7789091a8cf => generated 0 bytes in 635 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:52:36.817643 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16550d9-6023-4eb4-83b8-3349d1112c22 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:36.819649 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16550d9-6023-4eb4-83b8-3349d1112c22 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:36.820019 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16550d9-6023-4eb4-83b8-3349d1112c22 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:36.822360 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-eaff4757-695b-4c57-83ef-0a2aa4bae0e0 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:36.826482 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-eaff4757-695b-4c57-83ef-0a2aa4bae0e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:36.897850 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-c16550d9-6023-4eb4-83b8-3349d1112c22 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:36.903157 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-c16550d9-6023-4eb4-83b8-3349d1112c22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:36.921253 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:52:37.050292 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 829/1649] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 292 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:37.116351 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 821/1650] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 297 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:37.193272 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaff4757-695b-4c57-83ef-0a2aa4bae0e0 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:37.194730 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-eaff4757-695b-4c57-83ef-0a2aa4bae0e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:37.242556 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c16550d9-6023-4eb4-83b8-3349d1112c22 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:37.244491 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-c16550d9-6023-4eb4-83b8-3349d1112c22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:37.357720 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:52:37.380921 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c16550d9-6023-4eb4-83b8-3349d1112c22 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:37.394127 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-372d4697-1c8d-440a-9c44-010a6087a18a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:37.394727 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372d4697-1c8d-440a-9c44-010a6087a18a service neutron] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:37.395210 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372d4697-1c8d-440a-9c44-010a6087a18a service neutron] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:37.395324 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372d4697-1c8d-440a-9c44-010a6087a18a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:37.413997 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaff4757-695b-4c57-83ef-0a2aa4bae0e0 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:37.416961 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-372d4697-1c8d-440a-9c44-010a6087a18a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:37.418723 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-372d4697-1c8d-440a-9c44-010a6087a18a service neutron] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:37.425416 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2881a463-38a8-49cf-8ed0-e71d6b3755e9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:37.426087 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2881a463-38a8-49cf-8ed0-e71d6b3755e9 service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:37.426357 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2881a463-38a8-49cf-8ed0-e71d6b3755e9 service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:37.426636 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2881a463-38a8-49cf-8ed0-e71d6b3755e9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:37.440286 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 822/1651] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:52:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 198 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:52:37.442099 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-791b0eec-10e3-4097-98fb-518bf43d6447 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:37.442586 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-791b0eec-10e3-4097-98fb-518bf43d6447 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:37.444691 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-791b0eec-10e3-4097-98fb-518bf43d6447 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:37.450950 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2881a463-38a8-49cf-8ed0-e71d6b3755e9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:37.455582 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 823/1652] 10.222.0.27 () {54 vars in 962 bytes} [Sun Sep 11 00:52:37 2022] GET /identity => generated 270 bytes in 14 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:52:37.456296 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2881a463-38a8-49cf-8ed0-e71d6b3755e9 service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:37.458167 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ee37e4-7c0b-410e-b511-71b2bc7d18b8 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:37.458694 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ee37e4-7c0b-410e-b511-71b2bc7d18b8 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:37.459218 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ee37e4-7c0b-410e-b511-71b2bc7d18b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:37.484495 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 830/1653] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:52:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 292 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:52:37.488477 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-b1ee37e4-7c0b-410e-b511-71b2bc7d18b8 None None] MFA Rules not processed for user `835def0e6c6d4365a2767051509eb0d6`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:37.490022 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-b1ee37e4-7c0b-410e-b511-71b2bc7d18b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:37.716077 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 824/1654] 10.222.0.27 () {58 vars in 1059 bytes} [Sun Sep 11 00:52:37 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 259 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:37.719349 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0061fc-9d01-4e0e-bdde-163c1b0a3fa0 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:37.719650 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0061fc-9d01-4e0e-bdde-163c1b0a3fa0 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:37.719935 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0061fc-9d01-4e0e-bdde-163c1b0a3fa0 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:37.724575 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 831/1655] 10.222.0.27 () {54 vars in 912 bytes} [Sun Sep 11 00:52:37 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 11 00:52:37.736146 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3366e690-026d-4370-8ea6-1cf8fd03db9a None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:37.736814 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3366e690-026d-4370-8ea6-1cf8fd03db9a None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:37.737659 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3366e690-026d-4370-8ea6-1cf8fd03db9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:37.773041 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-3366e690-026d-4370-8ea6-1cf8fd03db9a None None] MFA Rules not processed for user `835def0e6c6d4365a2767051509eb0d6`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:37.774997 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-3366e690-026d-4370-8ea6-1cf8fd03db9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:37.799466 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 825/1656] 10.222.0.27 () {58 vars in 1009 bytes} [Sun Sep 11 00:52:37 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 66 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:37.811597 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c0061fc-9d01-4e0e-bdde-163c1b0a3fa0 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:37.814087 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-2c0061fc-9d01-4e0e-bdde-163c1b0a3fa0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:38.002719 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c0061fc-9d01-4e0e-bdde-163c1b0a3fa0 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:38.020245 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dea05be-ec16-490a-8e9f-de0221aafd22 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:38.025361 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dea05be-ec16-490a-8e9f-de0221aafd22 service cinder] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:38.026810 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dea05be-ec16-490a-8e9f-de0221aafd22 service cinder] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:38.027702 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dea05be-ec16-490a-8e9f-de0221aafd22 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:38.075489 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dea05be-ec16-490a-8e9f-de0221aafd22 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:38.084482 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dea05be-ec16-490a-8e9f-de0221aafd22 service cinder] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:38.147630 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 832/1657] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:52:37 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 336 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:52:39.540406 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3366e690-026d-4370-8ea6-1cf8fd03db9a None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:39.541923 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-3366e690-026d-4370-8ea6-1cf8fd03db9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:39.565215 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3366e690-026d-4370-8ea6-1cf8fd03db9a None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:39.579030 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dbad7c1-d59f-48be-a7bd-c5cc24209dcc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:39.579690 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dbad7c1-d59f-48be-a7bd-c5cc24209dcc admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:39.579969 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dbad7c1-d59f-48be-a7bd-c5cc24209dcc admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:39.580244 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dbad7c1-d59f-48be-a7bd-c5cc24209dcc admin admin] PATH_INFO: `/v3/projects/74b82ca3c7424f5197fe35282cbad2e6` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:39.584186 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dbad7c1-d59f-48be-a7bd-c5cc24209dcc admin admin] RBAC: Authorizing `identity:delete_project(project_id=74b82ca3c7424f5197fe35282cbad2e6)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:39.586152 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dbad7c1-d59f-48be-a7bd-c5cc24209dcc admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:39.668999 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:52:39.707150 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-2dbad7c1-d59f-48be-a7bd-c5cc24209dcc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '74b82ca3c7424f5197fe35282cbad2e6', 'request_id': 'req-2dbad7c1-d59f-48be-a7bd-c5cc24209dcc'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:39.728535 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-2dbad7c1-d59f-48be-a7bd-c5cc24209dcc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '74b82ca3c7424f5197fe35282cbad2e6', 'request_id': 'req-2dbad7c1-d59f-48be-a7bd-c5cc24209dcc'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:39.746532 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-2dbad7c1-d59f-48be-a7bd-c5cc24209dcc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '74b82ca3c7424f5197fe35282cbad2e6', 'request_id': 'req-2dbad7c1-d59f-48be-a7bd-c5cc24209dcc'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:39.778890 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-2dbad7c1-d59f-48be-a7bd-c5cc24209dcc admin admin] The token cache is being invalidate because project 74b82ca3c7424f5197fe35282cbad2e6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87923) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:52:39.779006 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-2dbad7c1-d59f-48be-a7bd-c5cc24209dcc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:39.783445 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-2dbad7c1-d59f-48be-a7bd-c5cc24209dcc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:39.788728 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 826/1658] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:52:39 2022] DELETE /identity/v3/projects/74b82ca3c7424f5197fe35282cbad2e6 => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:52:39.792602 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dea05be-ec16-490a-8e9f-de0221aafd22 service cinder] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:39.801852 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dbad7c1-d59f-48be-a7bd-c5cc24209dcc admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:39.808381 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-2dbad7c1-d59f-48be-a7bd-c5cc24209dcc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:39.824600 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dea05be-ec16-490a-8e9f-de0221aafd22 service cinder] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:40.016007 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8253194d-e868-4f56-a548-8b0a1584c860 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:40.016667 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8253194d-e868-4f56-a548-8b0a1584c860 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:40.016939 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8253194d-e868-4f56-a548-8b0a1584c860 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:40.017227 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8253194d-e868-4f56-a548-8b0a1584c860 admin admin] PATH_INFO: `/v3/users/5780ba3d9ec3462dab4bc731886be859` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:40.019642 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8253194d-e868-4f56-a548-8b0a1584c860 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5780ba3d9ec3462dab4bc731886be859)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:40.021590 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8253194d-e868-4f56-a548-8b0a1584c860 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:40.026052 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dbad7c1-d59f-48be-a7bd-c5cc24209dcc admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:40.036771 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3703353-0254-4e22-a9df-2e69b0c58535 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:40.037428 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3703353-0254-4e22-a9df-2e69b0c58535 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:40.037707 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3703353-0254-4e22-a9df-2e69b0c58535 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:40.037981 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3703353-0254-4e22-a9df-2e69b0c58535 admin admin] PATH_INFO: `/v3/projects/f4d1bc2ff0c7459ea7ef2aba844623ca` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:40.043586 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3703353-0254-4e22-a9df-2e69b0c58535 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f4d1bc2ff0c7459ea7ef2aba844623ca)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:40.048130 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3703353-0254-4e22-a9df-2e69b0c58535 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:40.110875 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-f3703353-0254-4e22-a9df-2e69b0c58535 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f4d1bc2ff0c7459ea7ef2aba844623ca', 'request_id': 'req-f3703353-0254-4e22-a9df-2e69b0c58535'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:40.121200 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-f3703353-0254-4e22-a9df-2e69b0c58535 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f4d1bc2ff0c7459ea7ef2aba844623ca', 'request_id': 'req-f3703353-0254-4e22-a9df-2e69b0c58535'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:40.127902 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-f3703353-0254-4e22-a9df-2e69b0c58535 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f4d1bc2ff0c7459ea7ef2aba844623ca', 'request_id': 'req-f3703353-0254-4e22-a9df-2e69b0c58535'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:40.153983 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-f3703353-0254-4e22-a9df-2e69b0c58535 admin admin] The token cache is being invalidate because project f4d1bc2ff0c7459ea7ef2aba844623ca was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87923) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:52:40.154492 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-8253194d-e868-4f56-a548-8b0a1584c860 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5780ba3d9ec3462dab4bc731886be859'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:40.154748 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-f3703353-0254-4e22-a9df-2e69b0c58535 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:40.156441 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-8253194d-e868-4f56-a548-8b0a1584c860 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5780ba3d9ec3462dab4bc731886be859'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:40.157107 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-f3703353-0254-4e22-a9df-2e69b0c58535 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:40.157231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-8253194d-e868-4f56-a548-8b0a1584c860 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5780ba3d9ec3462dab4bc731886be859'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:40.160569 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 827/1659] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:52:39 2022] DELETE /identity/v3/projects/f4d1bc2ff0c7459ea7ef2aba844623ca => generated 0 bytes in 359 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:52:40.161401 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3703353-0254-4e22-a9df-2e69b0c58535 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:40.165144 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-f3703353-0254-4e22-a9df-2e69b0c58535 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:40.176937 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-8253194d-e868-4f56-a548-8b0a1584c860 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5780ba3d9ec3462dab4bc731886be859'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:40.225838 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-8253194d-e868-4f56-a548-8b0a1584c860 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5780ba3d9ec3462dab4bc731886be859'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:40.255432 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:52:40.266622 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 833/1660] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:52:39 2022] DELETE /identity/v3/users/5780ba3d9ec3462dab4bc731886be859 => generated 0 bytes in 475 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:52:40.270862 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8253194d-e868-4f56-a548-8b0a1584c860 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:40.277774 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-8253194d-e868-4f56-a548-8b0a1584c860 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:40.307590 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:52:40.330889 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3703353-0254-4e22-a9df-2e69b0c58535 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:40.479002 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-615ff949-b078-492b-9422-6795cb68408a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:40.479659 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-615ff949-b078-492b-9422-6795cb68408a admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:40.480588 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-615ff949-b078-492b-9422-6795cb68408a admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:40.480873 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-615ff949-b078-492b-9422-6795cb68408a admin admin] PATH_INFO: `/v3/users/84cbd86bbf99414090fc704b9d0817bf` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:40.482074 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-615ff949-b078-492b-9422-6795cb68408a admin admin] RBAC: Authorizing `identity:delete_user(user_id=84cbd86bbf99414090fc704b9d0817bf)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:40.489657 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-615ff949-b078-492b-9422-6795cb68408a admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:40.503011 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8253194d-e868-4f56-a548-8b0a1584c860 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:40.521776 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28b62d30-3ec6-4d41-886d-ffcab9da9dbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:40.522860 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28b62d30-3ec6-4d41-886d-ffcab9da9dbe admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:40.523469 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28b62d30-3ec6-4d41-886d-ffcab9da9dbe admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:40.523902 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28b62d30-3ec6-4d41-886d-ffcab9da9dbe admin admin] PATH_INFO: `/v3/users/788e95ddb109462e81055cbd0653b99d` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:40.526952 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28b62d30-3ec6-4d41-886d-ffcab9da9dbe admin admin] RBAC: Authorizing `identity:delete_user(user_id=788e95ddb109462e81055cbd0653b99d)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:40.533517 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28b62d30-3ec6-4d41-886d-ffcab9da9dbe admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:40.588403 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-615ff949-b078-492b-9422-6795cb68408a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '84cbd86bbf99414090fc704b9d0817bf'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:40.590635 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-615ff949-b078-492b-9422-6795cb68408a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '84cbd86bbf99414090fc704b9d0817bf'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:40.591994 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-615ff949-b078-492b-9422-6795cb68408a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '84cbd86bbf99414090fc704b9d0817bf'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:40.617454 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-615ff949-b078-492b-9422-6795cb68408a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '84cbd86bbf99414090fc704b9d0817bf'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:40.658217 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-615ff949-b078-492b-9422-6795cb68408a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '84cbd86bbf99414090fc704b9d0817bf'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:40.664692 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-28b62d30-3ec6-4d41-886d-ffcab9da9dbe admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '788e95ddb109462e81055cbd0653b99d'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:40.665815 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-28b62d30-3ec6-4d41-886d-ffcab9da9dbe admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '788e95ddb109462e81055cbd0653b99d'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:40.666620 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-28b62d30-3ec6-4d41-886d-ffcab9da9dbe admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '788e95ddb109462e81055cbd0653b99d'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:40.685273 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 828/1661] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:52:40 2022] DELETE /identity/v3/users/84cbd86bbf99414090fc704b9d0817bf => generated 0 bytes in 521 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:52:40.690590 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-28b62d30-3ec6-4d41-886d-ffcab9da9dbe admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '788e95ddb109462e81055cbd0653b99d'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:40.694392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860b3b47-a57e-481b-92cd-2d5cecab2eae None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:40.694856 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860b3b47-a57e-481b-92cd-2d5cecab2eae None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:40.696026 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860b3b47-a57e-481b-92cd-2d5cecab2eae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:40.752770 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-28b62d30-3ec6-4d41-886d-ffcab9da9dbe admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '788e95ddb109462e81055cbd0653b99d'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:40.782930 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-860b3b47-a57e-481b-92cd-2d5cecab2eae None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:40.786148 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-860b3b47-a57e-481b-92cd-2d5cecab2eae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:40.797043 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 834/1662] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:52:40 2022] DELETE /identity/v3/users/788e95ddb109462e81055cbd0653b99d => generated 0 bytes in 530 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:52:40.963797 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 829/1663] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 272 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:41.161137 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28b62d30-3ec6-4d41-886d-ffcab9da9dbe admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:41.162782 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-28b62d30-3ec6-4d41-886d-ffcab9da9dbe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:41.305906 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28b62d30-3ec6-4d41-886d-ffcab9da9dbe admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:41.309819 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6a974e4-7f63-49fb-afed-8335001dbb16 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:41.310446 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a974e4-7f63-49fb-afed-8335001dbb16 service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:41.310728 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a974e4-7f63-49fb-afed-8335001dbb16 service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:41.311037 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a974e4-7f63-49fb-afed-8335001dbb16 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:41.404046 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:52:41.502253 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6a974e4-7f63-49fb-afed-8335001dbb16 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:41.503748 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6a974e4-7f63-49fb-afed-8335001dbb16 service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:41.537264 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 835/1664] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:52:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 377 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 11 00:52:41.763525 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b2f9fd-c4de-4463-8c0a-460cb1f1d14b None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:41.764348 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b2f9fd-c4de-4463-8c0a-460cb1f1d14b None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:41.766759 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b2f9fd-c4de-4463-8c0a-460cb1f1d14b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:41.888517 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-97b2f9fd-c4de-4463-8c0a-460cb1f1d14b None None] MFA Rules not processed for user `5df6979effd84ece8d75b631ba7da196`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:41.892790 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-97b2f9fd-c4de-4463-8c0a-460cb1f1d14b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:41.910491 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:52:42.080421 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6a974e4-7f63-49fb-afed-8335001dbb16 service neutron] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:42.083399 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-f6a974e4-7f63-49fb-afed-8335001dbb16 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:42.105507 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 830/1665] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:41 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 344 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:42.119888 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6a974e4-7f63-49fb-afed-8335001dbb16 service neutron] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:42.120464 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3f06a1-ae8a-496b-9643-63ed14e9e368 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:42.120737 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3f06a1-ae8a-496b-9643-63ed14e9e368 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:42.121010 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3f06a1-ae8a-496b-9643-63ed14e9e368 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:42.141997 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e918086a-7f42-4d3a-9f5e-89850e637634 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:42.144341 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e918086a-7f42-4d3a-9f5e-89850e637634 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:42.145820 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e918086a-7f42-4d3a-9f5e-89850e637634 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:42.147244 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e918086a-7f42-4d3a-9f5e-89850e637634 admin admin] PATH_INFO: `/v3/projects/5e8b0cb57b5c40dfbefb897953d81c7e` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:42.152538 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e918086a-7f42-4d3a-9f5e-89850e637634 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5e8b0cb57b5c40dfbefb897953d81c7e)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:42.172616 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: WARNING py.warnings [None req-e918086a-7f42-4d3a-9f5e-89850e637634 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 11 00:52:42.172616 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: warnings.warn(msg) Sep 11 00:52:42.172616 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: Sep 11 00:52:42.176019 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-8a3f06a1-ae8a-496b-9643-63ed14e9e368 None None] MFA Rules not processed for user `5df6979effd84ece8d75b631ba7da196`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:42.176309 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e918086a-7f42-4d3a-9f5e-89850e637634 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:42.177282 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-8a3f06a1-ae8a-496b-9643-63ed14e9e368 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:42.199192 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 831/1666] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:42 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:42.219610 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a3f06a1-ae8a-496b-9643-63ed14e9e368 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:42.221205 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-8a3f06a1-ae8a-496b-9643-63ed14e9e368 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:42.338636 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-e918086a-7f42-4d3a-9f5e-89850e637634 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5e8b0cb57b5c40dfbefb897953d81c7e', 'request_id': 'req-e918086a-7f42-4d3a-9f5e-89850e637634'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:42.359718 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-e918086a-7f42-4d3a-9f5e-89850e637634 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5e8b0cb57b5c40dfbefb897953d81c7e', 'request_id': 'req-e918086a-7f42-4d3a-9f5e-89850e637634'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:42.374016 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-e918086a-7f42-4d3a-9f5e-89850e637634 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5e8b0cb57b5c40dfbefb897953d81c7e', 'request_id': 'req-e918086a-7f42-4d3a-9f5e-89850e637634'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:42.402002 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-e918086a-7f42-4d3a-9f5e-89850e637634 admin admin] The token cache is being invalidate because project 5e8b0cb57b5c40dfbefb897953d81c7e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87924) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:52:42.403570 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-e918086a-7f42-4d3a-9f5e-89850e637634 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:42.406746 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-e918086a-7f42-4d3a-9f5e-89850e637634 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:42.415602 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 836/1667] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:52:42 2022] DELETE /identity/v3/projects/5e8b0cb57b5c40dfbefb897953d81c7e => generated 0 bytes in 332 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:52:42.421150 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d20e76-782a-4273-a78f-6405cd0b6975 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:42.421635 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d20e76-782a-4273-a78f-6405cd0b6975 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:42.422013 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d20e76-782a-4273-a78f-6405cd0b6975 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:42.461793 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-a2d20e76-782a-4273-a78f-6405cd0b6975 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:42.463190 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-a2d20e76-782a-4273-a78f-6405cd0b6975 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:42.463973 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a3f06a1-ae8a-496b-9643-63ed14e9e368 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:42.584159 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c365d9-aa22-4826-930b-a38401a4d225 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '835def0e6c6d4365a2767051509eb0d6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'ResellerAdmin', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:42.584946 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c365d9-aa22-4826-930b-a38401a4d225 service nova] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:42.585295 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c365d9-aa22-4826-930b-a38401a4d225 service nova] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:42.585581 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c365d9-aa22-4826-930b-a38401a4d225 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:42.588540 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-08c365d9-aa22-4826-930b-a38401a4d225 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:42.646089 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 837/1668] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 221 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:42.646089 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d20e76-782a-4273-a78f-6405cd0b6975 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:42.647864 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-a2d20e76-782a-4273-a78f-6405cd0b6975 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:42.648741 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:52:42.673049 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d20e76-782a-4273-a78f-6405cd0b6975 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:42.683799 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-babd4105-1767-410c-b5d8-eb97b85ad7c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:42.684869 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-babd4105-1767-410c-b5d8-eb97b85ad7c4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:42.684869 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-babd4105-1767-410c-b5d8-eb97b85ad7c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:42.685058 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-babd4105-1767-410c-b5d8-eb97b85ad7c4 admin admin] PATH_INFO: `/v3/projects/43c4afbc769248eb8c4f90c337a3d28b` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:42.687582 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-babd4105-1767-410c-b5d8-eb97b85ad7c4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=43c4afbc769248eb8c4f90c337a3d28b)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:42.689720 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-babd4105-1767-410c-b5d8-eb97b85ad7c4 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:42.714776 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c365d9-aa22-4826-930b-a38401a4d225 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:42.717059 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c365d9-aa22-4826-930b-a38401a4d225 service nova] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:42.736238 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 832/1669] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:52:42 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 518 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 11 00:52:42.737163 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c365d9-aa22-4826-930b-a38401a4d225 service nova] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:42.741107 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-08c365d9-aa22-4826-930b-a38401a4d225 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:42.766890 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c365d9-aa22-4826-930b-a38401a4d225 service nova] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:42.773535 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78ca74ba-4746-4fea-a9e4-bcc5b82c3916 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:42.775394 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ca74ba-4746-4fea-a9e4-bcc5b82c3916 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:42.776195 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ca74ba-4746-4fea-a9e4-bcc5b82c3916 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:42.776674 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ca74ba-4746-4fea-a9e4-bcc5b82c3916 admin admin] PATH_INFO: `/v3/projects/47ed436614e34447b8610dee2ffe06cc` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:42.778164 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78ca74ba-4746-4fea-a9e4-bcc5b82c3916 admin admin] RBAC: Authorizing `identity:delete_project(project_id=47ed436614e34447b8610dee2ffe06cc)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:42.781686 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78ca74ba-4746-4fea-a9e4-bcc5b82c3916 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:42.831690 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:52:42.833730 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-babd4105-1767-410c-b5d8-eb97b85ad7c4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '43c4afbc769248eb8c4f90c337a3d28b', 'request_id': 'req-babd4105-1767-410c-b5d8-eb97b85ad7c4'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:42.844551 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-babd4105-1767-410c-b5d8-eb97b85ad7c4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '43c4afbc769248eb8c4f90c337a3d28b', 'request_id': 'req-babd4105-1767-410c-b5d8-eb97b85ad7c4'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:42.857755 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-babd4105-1767-410c-b5d8-eb97b85ad7c4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '43c4afbc769248eb8c4f90c337a3d28b', 'request_id': 'req-babd4105-1767-410c-b5d8-eb97b85ad7c4'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:42.864957 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-78ca74ba-4746-4fea-a9e4-bcc5b82c3916 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '47ed436614e34447b8610dee2ffe06cc', 'request_id': 'req-78ca74ba-4746-4fea-a9e4-bcc5b82c3916'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:42.872774 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-78ca74ba-4746-4fea-a9e4-bcc5b82c3916 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '47ed436614e34447b8610dee2ffe06cc', 'request_id': 'req-78ca74ba-4746-4fea-a9e4-bcc5b82c3916'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:42.883025 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-78ca74ba-4746-4fea-a9e4-bcc5b82c3916 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '47ed436614e34447b8610dee2ffe06cc', 'request_id': 'req-78ca74ba-4746-4fea-a9e4-bcc5b82c3916'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:42.912354 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-78ca74ba-4746-4fea-a9e4-bcc5b82c3916 admin admin] The token cache is being invalidate because project 47ed436614e34447b8610dee2ffe06cc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87923) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:52:42.912888 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-78ca74ba-4746-4fea-a9e4-bcc5b82c3916 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:42.913213 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-babd4105-1767-410c-b5d8-eb97b85ad7c4 admin admin] The token cache is being invalidate because project 43c4afbc769248eb8c4f90c337a3d28b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87924) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:52:42.913896 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-babd4105-1767-410c-b5d8-eb97b85ad7c4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:42.914393 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-78ca74ba-4746-4fea-a9e4-bcc5b82c3916 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:42.915727 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-babd4105-1767-410c-b5d8-eb97b85ad7c4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:42.917677 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 833/1670] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:52:42 2022] DELETE /identity/v3/projects/47ed436614e34447b8610dee2ffe06cc => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:52:42.923318 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 838/1671] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:52:42 2022] DELETE /identity/v3/projects/43c4afbc769248eb8c4f90c337a3d28b => generated 0 bytes in 285 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:52:42.925841 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae457a99-3bf3-42d0-9afd-74cf736d0079 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:42.926611 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57f4dff-ecd5-45a6-8364-47450e6ba755 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:42.927137 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae457a99-3bf3-42d0-9afd-74cf736d0079 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:42.927550 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57f4dff-ecd5-45a6-8364-47450e6ba755 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:42.927950 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae457a99-3bf3-42d0-9afd-74cf736d0079 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:42.928325 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57f4dff-ecd5-45a6-8364-47450e6ba755 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:42.973649 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-c57f4dff-ecd5-45a6-8364-47450e6ba755 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:42.974589 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-ae457a99-3bf3-42d0-9afd-74cf736d0079 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:42.976192 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-c57f4dff-ecd5-45a6-8364-47450e6ba755 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:42.977594 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-ae457a99-3bf3-42d0-9afd-74cf736d0079 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:43.149964 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 839/1672] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 226 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:43.161328 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c57f4dff-ecd5-45a6-8364-47450e6ba755 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:43.162054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 834/1673] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 244 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:43.163642 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb8483b-91c2-433c-84a5-5094de264c72 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:43.163935 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb8483b-91c2-433c-84a5-5094de264c72 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:43.164208 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb8483b-91c2-433c-84a5-5094de264c72 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:43.168035 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-c57f4dff-ecd5-45a6-8364-47450e6ba755 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:43.195183 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c57f4dff-ecd5-45a6-8364-47450e6ba755 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:43.206057 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-3fb8483b-91c2-433c-84a5-5094de264c72 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:43.209531 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8c5585-df2a-4a2b-9fdc-d8b431065e7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:43.210293 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8c5585-df2a-4a2b-9fdc-d8b431065e7c admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:43.210584 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8c5585-df2a-4a2b-9fdc-d8b431065e7c admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:43.210883 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8c5585-df2a-4a2b-9fdc-d8b431065e7c admin admin] PATH_INFO: `/v3/users/27219727f89444e38cb27313860fbd11` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:43.211784 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-3fb8483b-91c2-433c-84a5-5094de264c72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:43.213282 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b8c5585-df2a-4a2b-9fdc-d8b431065e7c admin admin] RBAC: Authorizing `identity:delete_user(user_id=27219727f89444e38cb27313860fbd11)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:43.215256 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b8c5585-df2a-4a2b-9fdc-d8b431065e7c admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:43.234450 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 835/1674] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 71 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:43.238139 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb8483b-91c2-433c-84a5-5094de264c72 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:43.261038 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb8483b-91c2-433c-84a5-5094de264c72 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:43.267979 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8351e523-000d-4926-9195-ffad06998630 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:43.268864 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8351e523-000d-4926-9195-ffad06998630 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:43.270619 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8351e523-000d-4926-9195-ffad06998630 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:43.270967 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8351e523-000d-4926-9195-ffad06998630 admin admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:43.272261 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8351e523-000d-4926-9195-ffad06998630 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:52:43.273660 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8351e523-000d-4926-9195-ffad06998630 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:43.275733 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8351e523-000d-4926-9195-ffad06998630 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:43.304962 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 836/1675] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 00:52:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 70 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:52:43.305901 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8351e523-000d-4926-9195-ffad06998630 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:43.316158 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-8b8c5585-df2a-4a2b-9fdc-d8b431065e7c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '27219727f89444e38cb27313860fbd11'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:43.317229 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-8b8c5585-df2a-4a2b-9fdc-d8b431065e7c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '27219727f89444e38cb27313860fbd11'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:43.317891 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-8b8c5585-df2a-4a2b-9fdc-d8b431065e7c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '27219727f89444e38cb27313860fbd11'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:43.332732 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8351e523-000d-4926-9195-ffad06998630 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:43.333904 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-8b8c5585-df2a-4a2b-9fdc-d8b431065e7c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '27219727f89444e38cb27313860fbd11'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:43.345943 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aedc9ea-a41f-4c1e-b393-e4dd6ca7867d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:43.346559 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aedc9ea-a41f-4c1e-b393-e4dd6ca7867d admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:43.347685 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aedc9ea-a41f-4c1e-b393-e4dd6ca7867d admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:43.349336 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aedc9ea-a41f-4c1e-b393-e4dd6ca7867d admin admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:43.350004 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aedc9ea-a41f-4c1e-b393-e4dd6ca7867d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:52:43.350314 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aedc9ea-a41f-4c1e-b393-e4dd6ca7867d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:43.353907 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aedc9ea-a41f-4c1e-b393-e4dd6ca7867d admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:43.363678 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-8b8c5585-df2a-4a2b-9fdc-d8b431065e7c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '27219727f89444e38cb27313860fbd11'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:43.380970 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 837/1676] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 00:52:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 75 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:52:43.386533 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aedc9ea-a41f-4c1e-b393-e4dd6ca7867d admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:43.393405 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 840/1677] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:52:43 2022] DELETE /identity/v3/users/27219727f89444e38cb27313860fbd11 => generated 0 bytes in 240 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:52:43.393405 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8c5585-df2a-4a2b-9fdc-d8b431065e7c admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:43.393597 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-8b8c5585-df2a-4a2b-9fdc-d8b431065e7c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:43.395295 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-7aedc9ea-a41f-4c1e-b393-e4dd6ca7867d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:43.419742 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aedc9ea-a41f-4c1e-b393-e4dd6ca7867d admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:43.472349 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:52:43.516234 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=85,keystone:INSERT=2,keystone:DELETE=13 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:52:43.552667 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8c5585-df2a-4a2b-9fdc-d8b431065e7c admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:43.567761 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c954907-7803-495c-83f4-94380d2ce788 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:43.569369 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c954907-7803-495c-83f4-94380d2ce788 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:43.570134 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0bfb01a-6c6d-4300-8b24-cc0097bc220f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:43.570780 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c954907-7803-495c-83f4-94380d2ce788 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:43.571122 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c954907-7803-495c-83f4-94380d2ce788 admin admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:43.572530 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0bfb01a-6c6d-4300-8b24-cc0097bc220f admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:43.572821 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0bfb01a-6c6d-4300-8b24-cc0097bc220f admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:43.573099 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0bfb01a-6c6d-4300-8b24-cc0097bc220f admin admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:43.573765 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0bfb01a-6c6d-4300-8b24-cc0097bc220f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:43.574811 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c954907-7803-495c-83f4-94380d2ce788 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:43.578222 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0bfb01a-6c6d-4300-8b24-cc0097bc220f admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:43.579560 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c954907-7803-495c-83f4-94380d2ce788 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:43.657621 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 838/1678] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:52:43 2022] POST /identity/v3/projects => generated 355 bytes in 271 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:52:43.659548 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa4088a-3866-4ca0-8406-0a71cffafef1 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:43.660499 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa4088a-3866-4ca0-8406-0a71cffafef1 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:43.660800 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa4088a-3866-4ca0-8406-0a71cffafef1 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:43.668162 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 841/1679] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:52:43 2022] POST /identity/v3/projects => generated 379 bytes in 277 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:52:43.669082 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0bfb01a-6c6d-4300-8b24-cc0097bc220f admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:43.670186 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 839/1680] 10.222.0.27 () {54 vars in 912 bytes} [Sun Sep 11 00:52:43 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:52:43.671586 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa4088a-3866-4ca0-8406-0a71cffafef1 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:43.861342 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0bfb01a-6c6d-4300-8b24-cc0097bc220f admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:43.874829 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-225e1819-c03f-4b26-930e-55f7efdcdab2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:43.876841 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225e1819-c03f-4b26-930e-55f7efdcdab2 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:43.877125 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225e1819-c03f-4b26-930e-55f7efdcdab2 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:43.877395 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225e1819-c03f-4b26-930e-55f7efdcdab2 admin admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:43.878055 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-225e1819-c03f-4b26-930e-55f7efdcdab2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:43.881395 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-225e1819-c03f-4b26-930e-55f7efdcdab2 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:43.884780 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa4088a-3866-4ca0-8406-0a71cffafef1 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:43.897726 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1692c6d-e10b-444f-b45e-9306036b7f41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:43.898358 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1692c6d-e10b-444f-b45e-9306036b7f41 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:43.898630 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1692c6d-e10b-444f-b45e-9306036b7f41 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:43.898927 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1692c6d-e10b-444f-b45e-9306036b7f41 admin admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:43.900489 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1692c6d-e10b-444f-b45e-9306036b7f41 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:43.902621 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1692c6d-e10b-444f-b45e-9306036b7f41 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:43.932116 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 842/1681] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:52:43 2022] POST /identity/v3/users => generated 278 bytes in 263 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:52:43.933369 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95de0ef7-1053-4c41-9b33-5e73eb1ce491 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:43.933679 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95de0ef7-1053-4c41-9b33-5e73eb1ce491 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:43.933958 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95de0ef7-1053-4c41-9b33-5e73eb1ce491 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:43.950117 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 840/1682] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:52:43 2022] POST /identity/v3/users => generated 290 bytes in 279 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:52:43.951449 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1692c6d-e10b-444f-b45e-9306036b7f41 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:43.963364 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1692c6d-e10b-444f-b45e-9306036b7f41 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:43.969547 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cfdd3ea-799d-45db-a5a7-55edfdeb6a2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:43.970162 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cfdd3ea-799d-45db-a5a7-55edfdeb6a2e admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:43.971512 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cfdd3ea-799d-45db-a5a7-55edfdeb6a2e admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:43.971807 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cfdd3ea-799d-45db-a5a7-55edfdeb6a2e admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:43.972471 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cfdd3ea-799d-45db-a5a7-55edfdeb6a2e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:43.974095 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cfdd3ea-799d-45db-a5a7-55edfdeb6a2e admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:43.987142 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 841/1683] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:52:43 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:52:43.988804 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cfdd3ea-799d-45db-a5a7-55edfdeb6a2e admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:44.000572 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cfdd3ea-799d-45db-a5a7-55edfdeb6a2e admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:44.006701 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37bb3bac-2461-41e8-8c06-b40493d926e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:44.007373 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37bb3bac-2461-41e8-8c06-b40493d926e5 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:44.007637 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37bb3bac-2461-41e8-8c06-b40493d926e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:44.007907 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37bb3bac-2461-41e8-8c06-b40493d926e5 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:44.008562 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37bb3bac-2461-41e8-8c06-b40493d926e5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:44.011501 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37bb3bac-2461-41e8-8c06-b40493d926e5 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:44.026386 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 842/1684] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:52:43 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:52:44.027275 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37bb3bac-2461-41e8-8c06-b40493d926e5 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:44.040824 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37bb3bac-2461-41e8-8c06-b40493d926e5 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:44.046421 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-95de0ef7-1053-4c41-9b33-5e73eb1ce491 None None] MFA Rules not processed for user `a5f7b13283cb4ebfb25160aa6ad0e2f4`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:44.046865 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aca9237c-96ee-4b4d-ac11-a0d979e98528 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:44.047499 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca9237c-96ee-4b4d-ac11-a0d979e98528 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:44.047775 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca9237c-96ee-4b4d-ac11-a0d979e98528 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:44.047885 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-95de0ef7-1053-4c41-9b33-5e73eb1ce491 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:44.048091 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca9237c-96ee-4b4d-ac11-a0d979e98528 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:44.048688 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aca9237c-96ee-4b4d-ac11-a0d979e98528 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:44.050744 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aca9237c-96ee-4b4d-ac11-a0d979e98528 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:44.064142 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 843/1685] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:52:44 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:52:44.065028 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aca9237c-96ee-4b4d-ac11-a0d979e98528 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:44.078405 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aca9237c-96ee-4b4d-ac11-a0d979e98528 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:44.095507 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f11603-2d8e-433d-a04c-856a5b6b1368 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:44.096130 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f11603-2d8e-433d-a04c-856a5b6b1368 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:44.096402 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f11603-2d8e-433d-a04c-856a5b6b1368 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:44.096671 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f11603-2d8e-433d-a04c-856a5b6b1368 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:44.097319 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12f11603-2d8e-433d-a04c-856a5b6b1368 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:44.102487 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12f11603-2d8e-433d-a04c-856a5b6b1368 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:44.131452 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 844/1686] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:52:44 2022] GET /identity/v3/roles => generated 1420 bytes in 66 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:52:44.134456 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f11603-2d8e-433d-a04c-856a5b6b1368 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:44.147449 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 843/1687] 10.222.0.27 () {58 vars in 1009 bytes} [Sun Sep 11 00:52:43 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:44.148695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95de0ef7-1053-4c41-9b33-5e73eb1ce491 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:44.172708 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95de0ef7-1053-4c41-9b33-5e73eb1ce491 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:44.175732 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f11603-2d8e-433d-a04c-856a5b6b1368 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:44.180838 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6002fe0f-ae4b-41a2-aacc-380814412c63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:44.181608 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6002fe0f-ae4b-41a2-aacc-380814412c63 admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:44.181816 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6002fe0f-ae4b-41a2-aacc-380814412c63 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:44.182088 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6002fe0f-ae4b-41a2-aacc-380814412c63 admin admin] PATH_INFO: `/v3/projects/653a37f96cff4020a54b9efc0e393f33/users/c091fd8fee78476ca15fca9655597677/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:44.185816 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f434c88b-2da1-4cf8-8097-f73234198c55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:44.187154 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f434c88b-2da1-4cf8-8097-f73234198c55 admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:44.187611 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f434c88b-2da1-4cf8-8097-f73234198c55 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:44.187896 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f434c88b-2da1-4cf8-8097-f73234198c55 admin admin] PATH_INFO: `/v3/projects/d3c0ead056414321832a11081ed450ce/users/670c9b05847443219e040623b04e1499/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:44.204652 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6002fe0f-ae4b-41a2-aacc-380814412c63 admin admin] RBAC: Authorizing `identity:create_grant(project_id=653a37f96cff4020a54b9efc0e393f33, user_id=c091fd8fee78476ca15fca9655597677, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:44.207403 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6002fe0f-ae4b-41a2-aacc-380814412c63 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:44.216771 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f434c88b-2da1-4cf8-8097-f73234198c55 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d3c0ead056414321832a11081ed450ce, user_id=670c9b05847443219e040623b04e1499, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:44.221029 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f434c88b-2da1-4cf8-8097-f73234198c55 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:44.228107 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 845/1688] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:52:44 2022] PUT /identity/v3/projects/653a37f96cff4020a54b9efc0e393f33/users/c091fd8fee78476ca15fca9655597677/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 96 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:52:44.229286 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6002fe0f-ae4b-41a2-aacc-380814412c63 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:44.245517 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 844/1689] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:52:44 2022] PUT /identity/v3/projects/d3c0ead056414321832a11081ed450ce/users/670c9b05847443219e040623b04e1499/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 97 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:52:44.246294 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f434c88b-2da1-4cf8-8097-f73234198c55 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:44.247482 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-6002fe0f-ae4b-41a2-aacc-380814412c63 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:44.247995 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-f434c88b-2da1-4cf8-8097-f73234198c55 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:44.300298 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:52:44.335587 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f434c88b-2da1-4cf8-8097-f73234198c55 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:44.341094 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c07e4d51-f031-4a60-b595-e3041de346ed service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:44.343011 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07e4d51-f031-4a60-b595-e3041de346ed service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:44.343591 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07e4d51-f031-4a60-b595-e3041de346ed service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:44.343962 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07e4d51-f031-4a60-b595-e3041de346ed service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:44.352514 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:52:44.359268 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c07e4d51-f031-4a60-b595-e3041de346ed service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:44.360904 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c07e4d51-f031-4a60-b595-e3041de346ed service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:44.377292 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 845/1690] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:52:44 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:52:44.381401 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8f841e-a4c5-4557-b431-7526f498f91c None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:44.383159 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8f841e-a4c5-4557-b431-7526f498f91c None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:44.383971 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8f841e-a4c5-4557-b431-7526f498f91c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:44.425148 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6002fe0f-ae4b-41a2-aacc-380814412c63 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:44.463559 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-5c8f841e-a4c5-4557-b431-7526f498f91c None None] MFA Rules not processed for user `c091fd8fee78476ca15fca9655597677`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:44.464606 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-5c8f841e-a4c5-4557-b431-7526f498f91c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:44.567391 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c09bacf-6cb1-414c-9aca-f5d249ddfe49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:44.568052 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c09bacf-6cb1-414c-9aca-f5d249ddfe49 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:44.568332 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c09bacf-6cb1-414c-9aca-f5d249ddfe49 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:44.569084 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c09bacf-6cb1-414c-9aca-f5d249ddfe49 admin admin] PATH_INFO: `/v3/projects/d93d1761f7564b53883e3724fe85cc68` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:44.570290 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c09bacf-6cb1-414c-9aca-f5d249ddfe49 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d93d1761f7564b53883e3724fe85cc68)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:44.572290 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c09bacf-6cb1-414c-9aca-f5d249ddfe49 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:44.619365 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 846/1691] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:44 2022] POST /identity/v3/auth/tokens => generated 3232 bytes in 241 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:44.622101 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddf62f6-c0e1-4084-8f68-b79f09e25010 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:44.622397 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddf62f6-c0e1-4084-8f68-b79f09e25010 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:44.622671 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddf62f6-c0e1-4084-8f68-b79f09e25010 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:44.704244 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-7c09bacf-6cb1-414c-9aca-f5d249ddfe49 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd93d1761f7564b53883e3724fe85cc68', 'request_id': 'req-7c09bacf-6cb1-414c-9aca-f5d249ddfe49'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:44.707803 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-fddf62f6-c0e1-4084-8f68-b79f09e25010 None None] MFA Rules not processed for user `670c9b05847443219e040623b04e1499`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:44.709071 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-fddf62f6-c0e1-4084-8f68-b79f09e25010 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:44.724803 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-7c09bacf-6cb1-414c-9aca-f5d249ddfe49 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd93d1761f7564b53883e3724fe85cc68', 'request_id': 'req-7c09bacf-6cb1-414c-9aca-f5d249ddfe49'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:44.735057 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-7c09bacf-6cb1-414c-9aca-f5d249ddfe49 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd93d1761f7564b53883e3724fe85cc68', 'request_id': 'req-7c09bacf-6cb1-414c-9aca-f5d249ddfe49'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:44.766043 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-7c09bacf-6cb1-414c-9aca-f5d249ddfe49 admin admin] The token cache is being invalidate because project d93d1761f7564b53883e3724fe85cc68 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87923) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:52:44.766414 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-7c09bacf-6cb1-414c-9aca-f5d249ddfe49 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:44.768067 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-7c09bacf-6cb1-414c-9aca-f5d249ddfe49 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:44.770864 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 846/1692] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:52:44 2022] DELETE /identity/v3/projects/d93d1761f7564b53883e3724fe85cc68 => generated 0 bytes in 542 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:52:44.772856 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c09bacf-6cb1-414c-9aca-f5d249ddfe49 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:44.774997 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-7c09bacf-6cb1-414c-9aca-f5d249ddfe49 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:44.881435 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 847/1693] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:44 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 261 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:44.887387 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2af9c48-6f95-463f-aba7-d6bfcdb99106 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:44.887689 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2af9c48-6f95-463f-aba7-d6bfcdb99106 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:44.887965 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2af9c48-6f95-463f-aba7-d6bfcdb99106 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:44.955424 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-e2af9c48-6f95-463f-aba7-d6bfcdb99106 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:44.956668 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-e2af9c48-6f95-463f-aba7-d6bfcdb99106 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:44.964415 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c09bacf-6cb1-414c-9aca-f5d249ddfe49 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:44.973605 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e9fd43-3959-4769-a55f-98a437f5275c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:44.974248 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e9fd43-3959-4769-a55f-98a437f5275c admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:44.975229 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e9fd43-3959-4769-a55f-98a437f5275c admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:44.975565 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e9fd43-3959-4769-a55f-98a437f5275c admin admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:44.976229 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89e9fd43-3959-4769-a55f-98a437f5275c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:44.978393 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89e9fd43-3959-4769-a55f-98a437f5275c admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:44.990449 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 848/1694] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:44.990449 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2af9c48-6f95-463f-aba7-d6bfcdb99106 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:44.990449 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-e2af9c48-6f95-463f-aba7-d6bfcdb99106 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:45.011380 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:52:45.019828 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2af9c48-6f95-463f-aba7-d6bfcdb99106 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:45.035097 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a326edaa-16c3-4cdc-af17-d0a1296efc95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:45.035995 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a326edaa-16c3-4cdc-af17-d0a1296efc95 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:45.036279 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a326edaa-16c3-4cdc-af17-d0a1296efc95 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:45.036937 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a326edaa-16c3-4cdc-af17-d0a1296efc95 admin admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:45.037602 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a326edaa-16c3-4cdc-af17-d0a1296efc95 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:45.039580 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a326edaa-16c3-4cdc-af17-d0a1296efc95 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:45.044702 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 847/1695] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:52:44 2022] POST /identity/v3/projects => generated 353 bytes in 273 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:52:45.047088 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a26003-78ad-46db-aa1d-1c20a6f0e341 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:45.051327 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a26003-78ad-46db-aa1d-1c20a6f0e341 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:45.051853 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a26003-78ad-46db-aa1d-1c20a6f0e341 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:45.110577 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-42a26003-78ad-46db-aa1d-1c20a6f0e341 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:45.114875 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-42a26003-78ad-46db-aa1d-1c20a6f0e341 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:45.122039 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 849/1696] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:52:44 2022] POST /identity/v3/projects => generated 385 bytes in 134 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:52:45.123050 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a326edaa-16c3-4cdc-af17-d0a1296efc95 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:45.195839 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:52:45.272153 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 848/1697] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 227 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:45.273079 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a26003-78ad-46db-aa1d-1c20a6f0e341 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:45.284359 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a326edaa-16c3-4cdc-af17-d0a1296efc95 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:45.284991 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a26003-78ad-46db-aa1d-1c20a6f0e341 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:45.295092 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e3b70b1-8648-47e5-8660-fa9e0c3f76fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:45.295461 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a056d1-9cd9-4155-9b0e-1e4e62385961 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:45.296461 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3b70b1-8648-47e5-8660-fa9e0c3f76fd admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:45.296635 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a056d1-9cd9-4155-9b0e-1e4e62385961 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:45.296975 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3b70b1-8648-47e5-8660-fa9e0c3f76fd admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:45.297178 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a056d1-9cd9-4155-9b0e-1e4e62385961 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:45.297178 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a056d1-9cd9-4155-9b0e-1e4e62385961 admin admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:45.297797 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30a056d1-9cd9-4155-9b0e-1e4e62385961 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:45.299877 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30a056d1-9cd9-4155-9b0e-1e4e62385961 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:45.301917 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3b70b1-8648-47e5-8660-fa9e0c3f76fd admin admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:45.302674 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e3b70b1-8648-47e5-8660-fa9e0c3f76fd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:45.305768 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e3b70b1-8648-47e5-8660-fa9e0c3f76fd admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:45.350982 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 849/1698] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:52:45 2022] POST /identity/v3/users => generated 299 bytes in 76 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:52:45.353272 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a056d1-9cd9-4155-9b0e-1e4e62385961 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:45.363784 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 850/1699] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:52:45 2022] POST /identity/v3/users => generated 283 bytes in 241 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:52:45.364679 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e3b70b1-8648-47e5-8660-fa9e0c3f76fd admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:45.374351 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a056d1-9cd9-4155-9b0e-1e4e62385961 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:45.379509 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e3b70b1-8648-47e5-8660-fa9e0c3f76fd admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:45.384815 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2865cef-8ccf-49b1-8d1f-7cb3c873f4ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:45.385425 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2865cef-8ccf-49b1-8d1f-7cb3c873f4ba admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:45.385704 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2865cef-8ccf-49b1-8d1f-7cb3c873f4ba admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:45.385974 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2865cef-8ccf-49b1-8d1f-7cb3c873f4ba admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:45.386623 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2865cef-8ccf-49b1-8d1f-7cb3c873f4ba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:45.391262 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2865cef-8ccf-49b1-8d1f-7cb3c873f4ba admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:45.394000 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b1add1-24e4-494d-842c-f7fd03e9e354 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:45.396713 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b1add1-24e4-494d-842c-f7fd03e9e354 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:45.396997 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b1add1-24e4-494d-842c-f7fd03e9e354 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:45.397270 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b1add1-24e4-494d-842c-f7fd03e9e354 admin admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:45.397947 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54b1add1-24e4-494d-842c-f7fd03e9e354 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:52:45.398671 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54b1add1-24e4-494d-842c-f7fd03e9e354 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:45.406228 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54b1add1-24e4-494d-842c-f7fd03e9e354 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:45.406418 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 851/1700] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:52:45 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:52:45.408672 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2865cef-8ccf-49b1-8d1f-7cb3c873f4ba admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:45.438733 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2865cef-8ccf-49b1-8d1f-7cb3c873f4ba admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:45.448459 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 850/1701] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 00:52:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:52:45.448459 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b1add1-24e4-494d-842c-f7fd03e9e354 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:45.459757 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b88267-2879-45a3-8fd2-22055177f12e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:45.460716 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b88267-2879-45a3-8fd2-22055177f12e admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:45.460716 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b88267-2879-45a3-8fd2-22055177f12e admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:45.460953 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b88267-2879-45a3-8fd2-22055177f12e admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:45.461577 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47b88267-2879-45a3-8fd2-22055177f12e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:45.463659 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47b88267-2879-45a3-8fd2-22055177f12e admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:45.473657 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b1add1-24e4-494d-842c-f7fd03e9e354 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:45.485268 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a711104a-0b55-4dc2-9544-126b3d899ce0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:45.489484 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a711104a-0b55-4dc2-9544-126b3d899ce0 admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:45.489484 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a711104a-0b55-4dc2-9544-126b3d899ce0 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:45.489672 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a711104a-0b55-4dc2-9544-126b3d899ce0 admin admin] PATH_INFO: `/v3/projects/62dcebc4cf7b4f708957689493f34d55/users/1010109f1ef44b4eb7415b58c24741dc/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:45.497605 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 852/1702] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:52:45 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:52:45.497605 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b88267-2879-45a3-8fd2-22055177f12e admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:45.527292 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b88267-2879-45a3-8fd2-22055177f12e admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:45.530684 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a711104a-0b55-4dc2-9544-126b3d899ce0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=62dcebc4cf7b4f708957689493f34d55, user_id=1010109f1ef44b4eb7415b58c24741dc, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:45.538248 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d672e02c-10a7-4723-b80e-efb4031625a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:45.538909 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d672e02c-10a7-4723-b80e-efb4031625a0 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:45.539197 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d672e02c-10a7-4723-b80e-efb4031625a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:45.539469 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d672e02c-10a7-4723-b80e-efb4031625a0 admin admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:45.540176 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d672e02c-10a7-4723-b80e-efb4031625a0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:45.542908 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d672e02c-10a7-4723-b80e-efb4031625a0 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:45.553324 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a711104a-0b55-4dc2-9544-126b3d899ce0 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:45.597763 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 853/1703] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:52:45 2022] POST /identity/v3/projects => generated 371 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:52:45.597763 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d672e02c-10a7-4723-b80e-efb4031625a0 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:45.607591 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 851/1704] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:52:45 2022] PUT /identity/v3/projects/62dcebc4cf7b4f708957689493f34d55/users/1010109f1ef44b4eb7415b58c24741dc/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:52:45.609730 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a711104a-0b55-4dc2-9544-126b3d899ce0 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:45.748392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a711104a-0b55-4dc2-9544-126b3d899ce0 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:45.757864 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d672e02c-10a7-4723-b80e-efb4031625a0 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:45.759054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce47a544-b374-4fa9-bcbd-10dad19f5c59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:45.759695 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce47a544-b374-4fa9-bcbd-10dad19f5c59 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:45.759980 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce47a544-b374-4fa9-bcbd-10dad19f5c59 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:45.760311 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce47a544-b374-4fa9-bcbd-10dad19f5c59 admin admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:45.761086 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce47a544-b374-4fa9-bcbd-10dad19f5c59 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:45.763787 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce47a544-b374-4fa9-bcbd-10dad19f5c59 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:45.776049 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1b805a7-8152-4913-a135-d14b642068dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:45.776746 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b805a7-8152-4913-a135-d14b642068dd admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:45.777497 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b805a7-8152-4913-a135-d14b642068dd admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:45.777786 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b805a7-8152-4913-a135-d14b642068dd admin admin] PATH_INFO: `/v3/projects/cae42aac78b2408b9248ad5176669d3b/users/08879d8838be40f1b9ed5b59d4e48053/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:45.802407 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 852/1705] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:52:45 2022] POST /identity/v3/users => generated 286 bytes in 194 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:52:45.803726 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce47a544-b374-4fa9-bcbd-10dad19f5c59 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:45.806693 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1b805a7-8152-4913-a135-d14b642068dd admin admin] RBAC: Authorizing `identity:create_grant(project_id=cae42aac78b2408b9248ad5176669d3b, user_id=08879d8838be40f1b9ed5b59d4e48053, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:45.806847 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-ce47a544-b374-4fa9-bcbd-10dad19f5c59 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:45.813427 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1b805a7-8152-4913-a135-d14b642068dd admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:45.863098 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:52:45.864105 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 854/1706] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:52:45 2022] PUT /identity/v3/projects/cae42aac78b2408b9248ad5176669d3b/users/08879d8838be40f1b9ed5b59d4e48053/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 269 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:52:45.868420 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1b805a7-8152-4913-a135-d14b642068dd admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:45.929213 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce47a544-b374-4fa9-bcbd-10dad19f5c59 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:46.004516 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:52:46.011624 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1b805a7-8152-4913-a135-d14b642068dd admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:46.022220 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4750739c-ba00-44e6-8c21-37b875536349 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:46.023207 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4750739c-ba00-44e6-8c21-37b875536349 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:46.024126 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4750739c-ba00-44e6-8c21-37b875536349 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:46.024746 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4750739c-ba00-44e6-8c21-37b875536349 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:46.025856 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4750739c-ba00-44e6-8c21-37b875536349 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:46.027588 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf76e8cf-1585-4f0a-aa08-f752d75273dd service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cdbfc2ccd73e4a5eb8c999be5b094619', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:46.028216 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf76e8cf-1585-4f0a-aa08-f752d75273dd service placement] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:46.028493 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf76e8cf-1585-4f0a-aa08-f752d75273dd service placement] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:46.028760 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf76e8cf-1585-4f0a-aa08-f752d75273dd service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:46.030690 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4750739c-ba00-44e6-8c21-37b875536349 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:46.031642 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-bf76e8cf-1585-4f0a-aa08-f752d75273dd service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:46.048696 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf76e8cf-1585-4f0a-aa08-f752d75273dd service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:46.048696 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf76e8cf-1585-4f0a-aa08-f752d75273dd service placement] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:46.051641 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 855/1707] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:52:45 2022] GET /identity/v3/roles => generated 1420 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:52:46.053258 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd489ed2-891e-4acc-888f-805694878c5e None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:46.053755 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd489ed2-891e-4acc-888f-805694878c5e None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:46.054214 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd489ed2-891e-4acc-888f-805694878c5e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:46.073960 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 853/1708] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:52:45 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 271 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:52:46.075078 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf76e8cf-1585-4f0a-aa08-f752d75273dd service placement] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:46.086906 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-fd489ed2-891e-4acc-888f-805694878c5e None None] MFA Rules not processed for user `5df6979effd84ece8d75b631ba7da196`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:46.088566 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-fd489ed2-891e-4acc-888f-805694878c5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:46.098636 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf76e8cf-1585-4f0a-aa08-f752d75273dd service placement] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:46.113957 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca2822fb-c40e-4b05-8852-b52e0445560c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:46.114885 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2822fb-c40e-4b05-8852-b52e0445560c admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:46.115167 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2822fb-c40e-4b05-8852-b52e0445560c admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:46.119619 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2822fb-c40e-4b05-8852-b52e0445560c admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:46.120316 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca2822fb-c40e-4b05-8852-b52e0445560c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:46.121936 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca2822fb-c40e-4b05-8852-b52e0445560c admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:46.137938 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 854/1709] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:52:46 2022] GET /identity/v3/roles => generated 1420 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:52:46.138860 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca2822fb-c40e-4b05-8852-b52e0445560c admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:46.153485 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca2822fb-c40e-4b05-8852-b52e0445560c admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:46.162227 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a00039d3-5024-4e56-9e89-411cb7cb42d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:46.174548 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a00039d3-5024-4e56-9e89-411cb7cb42d6 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:46.176154 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a00039d3-5024-4e56-9e89-411cb7cb42d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:46.176491 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a00039d3-5024-4e56-9e89-411cb7cb42d6 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:46.179888 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a00039d3-5024-4e56-9e89-411cb7cb42d6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:46.181853 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a00039d3-5024-4e56-9e89-411cb7cb42d6 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:46.213572 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 855/1710] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:52:46 2022] GET /identity/v3/roles => generated 1420 bytes in 75 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:52:46.214593 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a00039d3-5024-4e56-9e89-411cb7cb42d6 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:46.222891 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-a00039d3-5024-4e56-9e89-411cb7cb42d6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:46.257807 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a00039d3-5024-4e56-9e89-411cb7cb42d6 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:46.261219 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 856/1711] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:46 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:46.261219 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd489ed2-891e-4acc-888f-805694878c5e None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:46.273520 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7614fda-324c-4c37-b2bb-48de1a241e86 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:46.275399 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7614fda-324c-4c37-b2bb-48de1a241e86 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:46.275767 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7614fda-324c-4c37-b2bb-48de1a241e86 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:46.276585 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7614fda-324c-4c37-b2bb-48de1a241e86 admin admin] PATH_INFO: `/v3/users/2b5c14fdcb014dcdae858a902812dd50` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:46.279710 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7614fda-324c-4c37-b2bb-48de1a241e86 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2b5c14fdcb014dcdae858a902812dd50)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:46.286467 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd489ed2-891e-4acc-888f-805694878c5e None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:46.287082 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7614fda-324c-4c37-b2bb-48de1a241e86 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:46.300977 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8845b8a-c38b-402e-9e0a-7306828cd09a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:46.301853 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8845b8a-c38b-402e-9e0a-7306828cd09a admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:46.302063 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8845b8a-c38b-402e-9e0a-7306828cd09a admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:46.302334 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8845b8a-c38b-402e-9e0a-7306828cd09a admin admin] PATH_INFO: `/v3/projects/62dcebc4cf7b4f708957689493f34d55/users/1010109f1ef44b4eb7415b58c24741dc/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:46.313058 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8845b8a-c38b-402e-9e0a-7306828cd09a admin admin] RBAC: Authorizing `identity:create_grant(project_id=62dcebc4cf7b4f708957689493f34d55, user_id=1010109f1ef44b4eb7415b58c24741dc, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:46.333116 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8845b8a-c38b-402e-9e0a-7306828cd09a admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:46.347355 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 857/1712] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:52:46 2022] PUT /identity/v3/projects/62dcebc4cf7b4f708957689493f34d55/users/1010109f1ef44b4eb7415b58c24741dc/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 87 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:52:46.348534 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8845b8a-c38b-402e-9e0a-7306828cd09a admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:46.503279 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-d7614fda-324c-4c37-b2bb-48de1a241e86 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2b5c14fdcb014dcdae858a902812dd50'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:46.504658 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-d7614fda-324c-4c37-b2bb-48de1a241e86 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2b5c14fdcb014dcdae858a902812dd50'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:46.507745 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-d7614fda-324c-4c37-b2bb-48de1a241e86 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2b5c14fdcb014dcdae858a902812dd50'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:46.531331 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-d7614fda-324c-4c37-b2bb-48de1a241e86 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2b5c14fdcb014dcdae858a902812dd50'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:46.550212 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8845b8a-c38b-402e-9e0a-7306828cd09a admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:46.568075 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca51a12d-b9fa-42c5-8da1-ec3c52165172 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:46.568714 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca51a12d-b9fa-42c5-8da1-ec3c52165172 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:46.568999 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca51a12d-b9fa-42c5-8da1-ec3c52165172 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:46.569194 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-d7614fda-324c-4c37-b2bb-48de1a241e86 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2b5c14fdcb014dcdae858a902812dd50'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:46.569297 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca51a12d-b9fa-42c5-8da1-ec3c52165172 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:46.569924 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca51a12d-b9fa-42c5-8da1-ec3c52165172 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:46.579392 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca51a12d-b9fa-42c5-8da1-ec3c52165172 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:46.604087 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 858/1714] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:52:46 2022] GET /identity/v3/roles => generated 1420 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:52:46.604357 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 856/1714] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:52:46 2022] DELETE /identity/v3/users/2b5c14fdcb014dcdae858a902812dd50 => generated 0 bytes in 387 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:52:46.604357 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9bd65d9-35b7-44ef-8f05-be141cf6d2e9 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:46.604357 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9bd65d9-35b7-44ef-8f05-be141cf6d2e9 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:46.604579 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca51a12d-b9fa-42c5-8da1-ec3c52165172 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:46.606670 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9bd65d9-35b7-44ef-8f05-be141cf6d2e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:46.612368 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-ca51a12d-b9fa-42c5-8da1-ec3c52165172 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:46.645305 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca51a12d-b9fa-42c5-8da1-ec3c52165172 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:46.674012 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:52:46.690343 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-a9bd65d9-35b7-44ef-8f05-be141cf6d2e9 None None] MFA Rules not processed for user `1010109f1ef44b4eb7415b58c24741dc`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:46.694203 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-a9bd65d9-35b7-44ef-8f05-be141cf6d2e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:46.831083 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e323b584-6efe-4da8-a14b-44e7695f2b3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:46.831595 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e323b584-6efe-4da8-a14b-44e7695f2b3b admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:46.831876 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e323b584-6efe-4da8-a14b-44e7695f2b3b admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:46.832159 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e323b584-6efe-4da8-a14b-44e7695f2b3b admin admin] PATH_INFO: `/v3/projects/cae42aac78b2408b9248ad5176669d3b/users/08879d8838be40f1b9ed5b59d4e48053/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:46.840622 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e323b584-6efe-4da8-a14b-44e7695f2b3b admin admin] RBAC: Authorizing `identity:create_grant(project_id=cae42aac78b2408b9248ad5176669d3b, user_id=08879d8838be40f1b9ed5b59d4e48053, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:46.844795 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e323b584-6efe-4da8-a14b-44e7695f2b3b admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:46.890365 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 859/1715] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:52:46 2022] PUT /identity/v3/projects/cae42aac78b2408b9248ad5176669d3b/users/08879d8838be40f1b9ed5b59d4e48053/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 279 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:52:46.890365 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0307a2b-0728-4ea1-82ac-13de997f53f0 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:46.890365 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0307a2b-0728-4ea1-82ac-13de997f53f0 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:46.890365 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0307a2b-0728-4ea1-82ac-13de997f53f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:46.939925 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-c0307a2b-0728-4ea1-82ac-13de997f53f0 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:46.941146 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-c0307a2b-0728-4ea1-82ac-13de997f53f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:46.977943 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:52:47.009334 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 857/1716] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:46 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 407 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:47.013718 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9bd65d9-35b7-44ef-8f05-be141cf6d2e9 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:47.023775 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-a9bd65d9-35b7-44ef-8f05-be141cf6d2e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:47.076545 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9bd65d9-35b7-44ef-8f05-be141cf6d2e9 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:47.161573 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 860/1717] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 277 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:47.165039 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c38c544c-eefe-4f75-9dd5-4e3f8fdbbce7 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:47.165334 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c38c544c-eefe-4f75-9dd5-4e3f8fdbbce7 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:47.165611 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c38c544c-eefe-4f75-9dd5-4e3f8fdbbce7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:47.259324 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-c38c544c-eefe-4f75-9dd5-4e3f8fdbbce7 None None] MFA Rules not processed for user `08879d8838be40f1b9ed5b59d4e48053`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:47.264142 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-c38c544c-eefe-4f75-9dd5-4e3f8fdbbce7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:47.266418 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb5f907-2d57-48c1-b753-fbf2fd8e64a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:47.267212 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb5f907-2d57-48c1-b753-fbf2fd8e64a4 admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:47.267562 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb5f907-2d57-48c1-b753-fbf2fd8e64a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:47.267933 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb5f907-2d57-48c1-b753-fbf2fd8e64a4 admin admin] PATH_INFO: `/v3/projects/47891decc6d14bffb70bb4fb7c46c458/users/9048531ea3c64fe793aec2de9b56b2bf/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:47.330920 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecb5f907-2d57-48c1-b753-fbf2fd8e64a4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=47891decc6d14bffb70bb4fb7c46c458, user_id=9048531ea3c64fe793aec2de9b56b2bf, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:47.334519 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecb5f907-2d57-48c1-b753-fbf2fd8e64a4 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:47.405784 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 858/1718] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:52:47 2022] PUT /identity/v3/projects/47891decc6d14bffb70bb4fb7c46c458/users/9048531ea3c64fe793aec2de9b56b2bf/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 393 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:52:47.407210 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb5f907-2d57-48c1-b753-fbf2fd8e64a4 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:47.414660 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-ecb5f907-2d57-48c1-b753-fbf2fd8e64a4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:47.488725 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 861/1719] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:47 2022] POST /identity/v3/auth/tokens => generated 3297 bytes in 326 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:47.488725 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c38c544c-eefe-4f75-9dd5-4e3f8fdbbce7 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:47.494993 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-c38c544c-eefe-4f75-9dd5-4e3f8fdbbce7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:47.637282 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb5f907-2d57-48c1-b753-fbf2fd8e64a4 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:47.651633 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59749877-27e6-47ef-b727-e7ba899d7fe3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:47.652276 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59749877-27e6-47ef-b727-e7ba899d7fe3 service cinder] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:47.652555 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59749877-27e6-47ef-b727-e7ba899d7fe3 service cinder] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:47.652827 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59749877-27e6-47ef-b727-e7ba899d7fe3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:47.670176 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c38c544c-eefe-4f75-9dd5-4e3f8fdbbce7 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:47.735315 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:52:47.796433 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59749877-27e6-47ef-b727-e7ba899d7fe3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:47.798060 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59749877-27e6-47ef-b727-e7ba899d7fe3 service cinder] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:47.853397 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 859/1720] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:52:47 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 450 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:52:47.855036 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e19353-2afd-4d2c-a83a-cc729fcbe92c None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:47.855331 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e19353-2afd-4d2c-a83a-cc729fcbe92c None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:47.855609 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e19353-2afd-4d2c-a83a-cc729fcbe92c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:47.894617 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029d4074-b084-4ce3-a389-150a6485776f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:47.895654 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029d4074-b084-4ce3-a389-150a6485776f service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:47.896146 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029d4074-b084-4ce3-a389-150a6485776f service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:47.896600 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029d4074-b084-4ce3-a389-150a6485776f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:47.898721 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-60e19353-2afd-4d2c-a83a-cc729fcbe92c None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:47.899960 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-60e19353-2afd-4d2c-a83a-cc729fcbe92c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:47.932724 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:52:48.134879 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 860/1721] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 281 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:48.138050 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e19353-2afd-4d2c-a83a-cc729fcbe92c None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:48.140714 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-60e19353-2afd-4d2c-a83a-cc729fcbe92c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:48.145794 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029d4074-b084-4ce3-a389-150a6485776f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:48.147303 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029d4074-b084-4ce3-a389-150a6485776f service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:48.173356 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 862/1722] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:52:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 687 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:52:48.173762 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029d4074-b084-4ce3-a389-150a6485776f service neutron] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:48.181296 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e19353-2afd-4d2c-a83a-cc729fcbe92c None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:48.185125 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-029d4074-b084-4ce3-a389-150a6485776f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:48.189864 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1d550d-b4c4-4b54-a9c1-3344b5c5d9ad service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:48.190809 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1d550d-b4c4-4b54-a9c1-3344b5c5d9ad service cinder] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:48.193786 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1d550d-b4c4-4b54-a9c1-3344b5c5d9ad service cinder] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:48.194329 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1d550d-b4c4-4b54-a9c1-3344b5c5d9ad service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:48.217575 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029d4074-b084-4ce3-a389-150a6485776f service neutron] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:48.247056 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-091b4f8f-12bc-42d8-ab6d-c1262d5fce99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:48.248031 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091b4f8f-12bc-42d8-ab6d-c1262d5fce99 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:48.248506 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091b4f8f-12bc-42d8-ab6d-c1262d5fce99 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:48.249444 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091b4f8f-12bc-42d8-ab6d-c1262d5fce99 admin admin] PATH_INFO: `/v3/users/670c9b05847443219e040623b04e1499` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:48.252345 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-091b4f8f-12bc-42d8-ab6d-c1262d5fce99 admin admin] RBAC: Authorizing `identity:delete_user(user_id=670c9b05847443219e040623b04e1499)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:48.255793 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-091b4f8f-12bc-42d8-ab6d-c1262d5fce99 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:48.345272 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca1d550d-b4c4-4b54-a9c1-3344b5c5d9ad service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:48.347460 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca1d550d-b4c4-4b54-a9c1-3344b5c5d9ad service cinder] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:48.393033 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 861/1723] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:52:48 2022] GET /identity/v3/auth/tokens => generated 3297 bytes in 257 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:52:48.396585 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1d550d-b4c4-4b54-a9c1-3344b5c5d9ad service cinder] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:48.399733 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-091b4f8f-12bc-42d8-ab6d-c1262d5fce99 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '670c9b05847443219e040623b04e1499'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:48.401754 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-ca1d550d-b4c4-4b54-a9c1-3344b5c5d9ad service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:48.402877 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-091b4f8f-12bc-42d8-ab6d-c1262d5fce99 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '670c9b05847443219e040623b04e1499'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:48.404220 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-091b4f8f-12bc-42d8-ab6d-c1262d5fce99 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '670c9b05847443219e040623b04e1499'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:48.428526 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-091b4f8f-12bc-42d8-ab6d-c1262d5fce99 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '670c9b05847443219e040623b04e1499'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:48.455780 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1d550d-b4c4-4b54-a9c1-3344b5c5d9ad service cinder] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:48.465902 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-091b4f8f-12bc-42d8-ab6d-c1262d5fce99 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '670c9b05847443219e040623b04e1499'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:48.478474 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f592bb7-91dd-4440-9227-b1499e3e3009 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:48.482635 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f592bb7-91dd-4440-9227-b1499e3e3009 service neutron] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:48.485881 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f592bb7-91dd-4440-9227-b1499e3e3009 service neutron] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:48.486556 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f592bb7-91dd-4440-9227-b1499e3e3009 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:48.494295 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-7f592bb7-91dd-4440-9227-b1499e3e3009 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:48.494913 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 863/1724] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:52:48 2022] DELETE /identity/v3/users/670c9b05847443219e040623b04e1499 => generated 0 bytes in 318 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:52:48.496294 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee52f82f-63c6-4bf6-810a-b95048343047 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:48.496589 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee52f82f-63c6-4bf6-810a-b95048343047 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:48.496869 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee52f82f-63c6-4bf6-810a-b95048343047 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:48.533982 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f592bb7-91dd-4440-9227-b1499e3e3009 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:48.547907 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f592bb7-91dd-4440-9227-b1499e3e3009 service neutron] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:48.548111 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-ee52f82f-63c6-4bf6-810a-b95048343047 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:48.549374 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-ee52f82f-63c6-4bf6-810a-b95048343047 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:48.585533 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 862/1725] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:52:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 192 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:52:48.586406 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f592bb7-91dd-4440-9227-b1499e3e3009 service neutron] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:48.590450 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-7f592bb7-91dd-4440-9227-b1499e3e3009 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:48.623075 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f592bb7-91dd-4440-9227-b1499e3e3009 service neutron] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:48.737936 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 864/1726] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 242 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:48.766246 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4547c70d-86c8-4d85-85a4-6a9ea1b4f15b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:48.768036 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4547c70d-86c8-4d85-85a4-6a9ea1b4f15b service cinder] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:48.768318 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4547c70d-86c8-4d85-85a4-6a9ea1b4f15b service cinder] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:48.768595 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4547c70d-86c8-4d85-85a4-6a9ea1b4f15b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:48.773836 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-4547c70d-86c8-4d85-85a4-6a9ea1b4f15b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:48.791819 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:52:48.829901 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee52f82f-63c6-4bf6-810a-b95048343047 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:48.832633 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-ee52f82f-63c6-4bf6-810a-b95048343047 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:48.924870 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:52:49.026201 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4547c70d-86c8-4d85-85a4-6a9ea1b4f15b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:49.028767 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4547c70d-86c8-4d85-85a4-6a9ea1b4f15b service cinder] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:49.059159 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee52f82f-63c6-4bf6-810a-b95048343047 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:49.062200 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 863/1727] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:52:48 2022] GET /identity/v3/auth/tokens => generated 3232 bytes in 476 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:52:49.085653 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-744123f2-7a08-4def-a2ba-693f4ea84879 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:49.086530 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744123f2-7a08-4def-a2ba-693f4ea84879 service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:49.086839 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744123f2-7a08-4def-a2ba-693f4ea84879 service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:49.087283 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744123f2-7a08-4def-a2ba-693f4ea84879 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:49.091941 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4547c70d-86c8-4d85-85a4-6a9ea1b4f15b service cinder] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:49.096305 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-4547c70d-86c8-4d85-85a4-6a9ea1b4f15b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:49.149580 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-744123f2-7a08-4def-a2ba-693f4ea84879 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:49.151165 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-744123f2-7a08-4def-a2ba-693f4ea84879 service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:49.153687 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4547c70d-86c8-4d85-85a4-6a9ea1b4f15b service cinder] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:49.174507 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001ddb4c-b1ae-4294-a708-bffc39b7786f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:49.175830 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 865/1728] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:52:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 346 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 11 00:52:49.179825 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001ddb4c-b1ae-4294-a708-bffc39b7786f admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:49.180117 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001ddb4c-b1ae-4294-a708-bffc39b7786f admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:49.180400 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001ddb4c-b1ae-4294-a708-bffc39b7786f admin admin] PATH_INFO: `/v3/projects/93b2f717c20042d0809f2c51a658fb83` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:49.185651 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-001ddb4c-b1ae-4294-a708-bffc39b7786f admin admin] RBAC: Authorizing `identity:delete_project(project_id=93b2f717c20042d0809f2c51a658fb83)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:49.191482 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-001ddb4c-b1ae-4294-a708-bffc39b7786f admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:49.324042 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4a0d5e-c66d-47ba-afde-daf94484717a None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:49.324551 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4a0d5e-c66d-47ba-afde-daf94484717a None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:49.325067 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4a0d5e-c66d-47ba-afde-daf94484717a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:49.366102 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-001ddb4c-b1ae-4294-a708-bffc39b7786f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '93b2f717c20042d0809f2c51a658fb83', 'request_id': 'req-001ddb4c-b1ae-4294-a708-bffc39b7786f'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:49.388184 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-001ddb4c-b1ae-4294-a708-bffc39b7786f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '93b2f717c20042d0809f2c51a658fb83', 'request_id': 'req-001ddb4c-b1ae-4294-a708-bffc39b7786f'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:49.406031 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-001ddb4c-b1ae-4294-a708-bffc39b7786f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '93b2f717c20042d0809f2c51a658fb83', 'request_id': 'req-001ddb4c-b1ae-4294-a708-bffc39b7786f'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:49.430833 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-5f4a0d5e-c66d-47ba-afde-daf94484717a None None] MFA Rules not processed for user `73a3d3c48913474ab8fa8fedd1bc7d03`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:49.435715 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-5f4a0d5e-c66d-47ba-afde-daf94484717a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:49.458108 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-001ddb4c-b1ae-4294-a708-bffc39b7786f admin admin] The token cache is being invalidate because project 93b2f717c20042d0809f2c51a658fb83 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87923) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:52:49.458940 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-001ddb4c-b1ae-4294-a708-bffc39b7786f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:49.461105 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-001ddb4c-b1ae-4294-a708-bffc39b7786f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:49.471553 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 864/1729] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:52:49 2022] DELETE /identity/v3/projects/93b2f717c20042d0809f2c51a658fb83 => generated 0 bytes in 376 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:52:49.473799 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae9d709-0779-4a47-9662-2721efcb7779 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:49.474093 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae9d709-0779-4a47-9662-2721efcb7779 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:49.477688 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae9d709-0779-4a47-9662-2721efcb7779 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:49.534928 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-1ae9d709-0779-4a47-9662-2721efcb7779 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:49.536548 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-1ae9d709-0779-4a47-9662-2721efcb7779 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:49.630045 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 866/1730] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:49 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 307 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:52:49.637020 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f4a0d5e-c66d-47ba-afde-daf94484717a None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:49.638749 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-5f4a0d5e-c66d-47ba-afde-daf94484717a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:49.774669 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 865/1731] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 302 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:49.776154 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a193dc-1c92-493e-896a-bca2e3c62f11 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:49.776437 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a193dc-1c92-493e-896a-bca2e3c62f11 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:49.776706 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a193dc-1c92-493e-896a-bca2e3c62f11 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:49.788645 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f4a0d5e-c66d-47ba-afde-daf94484717a None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:49.805078 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f409537c-5121-42c1-9872-9f6b07dc1b19 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '835def0e6c6d4365a2767051509eb0d6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'ResellerAdmin', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:49.807244 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f409537c-5121-42c1-9872-9f6b07dc1b19 service nova] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:49.807544 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f409537c-5121-42c1-9872-9f6b07dc1b19 service nova] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:49.807824 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f409537c-5121-42c1-9872-9f6b07dc1b19 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:49.833034 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f409537c-5121-42c1-9872-9f6b07dc1b19 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:49.837773 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-b8a193dc-1c92-493e-896a-bca2e3c62f11 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:49.839354 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-b8a193dc-1c92-493e-896a-bca2e3c62f11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:49.840029 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f409537c-5121-42c1-9872-9f6b07dc1b19 service nova] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:49.869083 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 866/1732] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:49.873401 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a193dc-1c92-493e-896a-bca2e3c62f11 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:49.900055 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a193dc-1c92-493e-896a-bca2e3c62f11 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:49.903941 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 867/1733] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:52:49 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 267 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:52:49.907610 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-424be9d1-f135-4d92-b893-ec97b4d4b3a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:49.908586 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424be9d1-f135-4d92-b893-ec97b4d4b3a3 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:49.909045 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424be9d1-f135-4d92-b893-ec97b4d4b3a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:49.909459 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424be9d1-f135-4d92-b893-ec97b4d4b3a3 admin admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:49.910529 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-424be9d1-f135-4d92-b893-ec97b4d4b3a3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:52:49.912077 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-424be9d1-f135-4d92-b893-ec97b4d4b3a3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:49.913844 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-424be9d1-f135-4d92-b893-ec97b4d4b3a3 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:49.947750 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f409537c-5121-42c1-9872-9f6b07dc1b19 service nova] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:49.956091 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 867/1734] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 00:52:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:52:49.972224 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f409537c-5121-42c1-9872-9f6b07dc1b19 service nova] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:49.992371 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b4d0b77-c649-4533-a72e-3dd73b9572ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:49.993006 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4d0b77-c649-4533-a72e-3dd73b9572ac admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:49.993338 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4d0b77-c649-4533-a72e-3dd73b9572ac admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:49.993754 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4d0b77-c649-4533-a72e-3dd73b9572ac admin admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:49.996394 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b4d0b77-c649-4533-a72e-3dd73b9572ac admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:50.004998 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b4d0b77-c649-4533-a72e-3dd73b9572ac admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:50.136562 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 868/1735] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:52:49 2022] POST /identity/v3/projects => generated 405 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:52:50.141525 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-424be9d1-f135-4d92-b893-ec97b4d4b3a3 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:50.161659 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-424be9d1-f135-4d92-b893-ec97b4d4b3a3 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:50.175121 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:52:50.319012 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9dd7ad0-da44-4ca7-b8d6-832cb3688c65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:50.319778 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9dd7ad0-da44-4ca7-b8d6-832cb3688c65 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:50.320121 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9dd7ad0-da44-4ca7-b8d6-832cb3688c65 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:50.320461 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9dd7ad0-da44-4ca7-b8d6-832cb3688c65 admin admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:50.321231 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9dd7ad0-da44-4ca7-b8d6-832cb3688c65 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:50.326151 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9dd7ad0-da44-4ca7-b8d6-832cb3688c65 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:50.376637 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 868/1736] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:52:50 2022] POST /identity/v3/users => generated 303 bytes in 236 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:52:50.380379 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b4d0b77-c649-4533-a72e-3dd73b9572ac admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:50.402528 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b4d0b77-c649-4533-a72e-3dd73b9572ac admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:50.409150 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b50c90ba-7821-4ef2-a3ac-564c45c2d44e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:50.409787 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b50c90ba-7821-4ef2-a3ac-564c45c2d44e admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:50.410876 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b50c90ba-7821-4ef2-a3ac-564c45c2d44e admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:50.410876 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b50c90ba-7821-4ef2-a3ac-564c45c2d44e admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:50.411311 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b50c90ba-7821-4ef2-a3ac-564c45c2d44e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:50.412958 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b50c90ba-7821-4ef2-a3ac-564c45c2d44e admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:50.439087 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 869/1737] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:52:50 2022] GET /identity/v3/roles => generated 1420 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:52:50.442453 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9dd7ad0-da44-4ca7-b8d6-832cb3688c65 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:50.465648 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9dd7ad0-da44-4ca7-b8d6-832cb3688c65 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:50.476673 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-247b5e33-f3aa-4bac-9aa6-38fac534c637 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:50.477558 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247b5e33-f3aa-4bac-9aa6-38fac534c637 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:50.478066 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247b5e33-f3aa-4bac-9aa6-38fac534c637 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:50.478485 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247b5e33-f3aa-4bac-9aa6-38fac534c637 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:50.479458 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-247b5e33-f3aa-4bac-9aa6-38fac534c637 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:50.481346 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-247b5e33-f3aa-4bac-9aa6-38fac534c637 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:50.525105 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08757ac0-9f9f-4cce-8d5a-75f4761489df None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:50.525625 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08757ac0-9f9f-4cce-8d5a-75f4761489df None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:50.526679 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08757ac0-9f9f-4cce-8d5a-75f4761489df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:50.547029 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 869/1738] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:52:50 2022] GET /identity/v3/roles => generated 1420 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:52:50.553496 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:52:50.558822 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-247b5e33-f3aa-4bac-9aa6-38fac534c637 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:50.608048 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-247b5e33-f3aa-4bac-9aa6-38fac534c637 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:50.629195 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f1bdac-85ea-4403-82b2-68162f54c4f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:50.645899 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f1bdac-85ea-4403-82b2-68162f54c4f3 admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:50.646102 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f1bdac-85ea-4403-82b2-68162f54c4f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:50.646374 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f1bdac-85ea-4403-82b2-68162f54c4f3 admin admin] PATH_INFO: `/v3/projects/0cf1b65c9a0545f7a009c34c8f8175c3/users/6b712be4c60a46e591301698da7f13dc/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:50.676781 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-08757ac0-9f9f-4cce-8d5a-75f4761489df None None] MFA Rules not processed for user `9fc4274836844ae4ad5d16d74c36f671`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:50.678735 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-08757ac0-9f9f-4cce-8d5a-75f4761489df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:50.696206 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33f1bdac-85ea-4403-82b2-68162f54c4f3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0cf1b65c9a0545f7a009c34c8f8175c3, user_id=6b712be4c60a46e591301698da7f13dc, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:50.704616 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33f1bdac-85ea-4403-82b2-68162f54c4f3 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:50.756913 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 870/1739] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:52:50 2022] PUT /identity/v3/projects/0cf1b65c9a0545f7a009c34c8f8175c3/users/6b712be4c60a46e591301698da7f13dc/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:52:50.760205 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864f13ac-bd22-4add-98b3-d629c3642429 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:50.763159 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864f13ac-bd22-4add-98b3-d629c3642429 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:50.766384 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864f13ac-bd22-4add-98b3-d629c3642429 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:50.829661 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 870/1740] 10.222.0.27 () {58 vars in 1009 bytes} [Sun Sep 11 00:52:50 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 306 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:50.895704 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-864f13ac-bd22-4add-98b3-d629c3642429 None None] MFA Rules not processed for user `6b712be4c60a46e591301698da7f13dc`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:50.903450 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-864f13ac-bd22-4add-98b3-d629c3642429 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:51.095977 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 871/1741] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:50 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 338 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:51.099330 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08757ac0-9f9f-4cce-8d5a-75f4761489df None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:51.101084 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-08757ac0-9f9f-4cce-8d5a-75f4761489df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:51.106251 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864f13ac-bd22-4add-98b3-d629c3642429 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:51.257534 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:52:51.272744 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864f13ac-bd22-4add-98b3-d629c3642429 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:51.285439 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7791ca6b-e6fb-4410-abeb-ee318bd81478 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:51.286330 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7791ca6b-e6fb-4410-abeb-ee318bd81478 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:51.287646 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7791ca6b-e6fb-4410-abeb-ee318bd81478 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:51.287646 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7791ca6b-e6fb-4410-abeb-ee318bd81478 admin admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:51.289430 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7791ca6b-e6fb-4410-abeb-ee318bd81478 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:51.293021 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7791ca6b-e6fb-4410-abeb-ee318bd81478 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:51.345118 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 872/1742] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:52:51 2022] POST /identity/v3/projects => generated 403 bytes in 239 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:52:51.348402 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08757ac0-9f9f-4cce-8d5a-75f4761489df None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:51.349101 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7791ca6b-e6fb-4410-abeb-ee318bd81478 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:51.486448 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:52:51.491915 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7791ca6b-e6fb-4410-abeb-ee318bd81478 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:51.500545 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f30045-a82b-4e1a-8e01-a10821ddbf74 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bcb73dec73464a3abc4b774abb4f416f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:51.501191 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f30045-a82b-4e1a-8e01-a10821ddbf74 service swift] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:51.501475 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f30045-a82b-4e1a-8e01-a10821ddbf74 service swift] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:51.501748 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f30045-a82b-4e1a-8e01-a10821ddbf74 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:51.506639 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bc708da-e21b-4724-8c8b-7422af807f34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:51.507293 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc708da-e21b-4724-8c8b-7422af807f34 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:51.507572 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc708da-e21b-4724-8c8b-7422af807f34 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:51.507845 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc708da-e21b-4724-8c8b-7422af807f34 admin admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:51.508494 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bc708da-e21b-4724-8c8b-7422af807f34 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:51.517922 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bc708da-e21b-4724-8c8b-7422af807f34 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:51.585199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 873/1743] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:52:51 2022] POST /identity/v3/users => generated 308 bytes in 234 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:52:51.586542 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bc708da-e21b-4724-8c8b-7422af807f34 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:51.612891 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bc708da-e21b-4724-8c8b-7422af807f34 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:51.625984 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b825e2cb-20f4-4e72-b52f-3e1857b3840a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:51.626805 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b825e2cb-20f4-4e72-b52f-3e1857b3840a admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:51.627097 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b825e2cb-20f4-4e72-b52f-3e1857b3840a admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:51.627372 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b825e2cb-20f4-4e72-b52f-3e1857b3840a admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:51.628032 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b825e2cb-20f4-4e72-b52f-3e1857b3840a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:51.640247 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b825e2cb-20f4-4e72-b52f-3e1857b3840a admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:51.689159 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 874/1744] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:52:51 2022] GET /identity/v3/roles => generated 1420 bytes in 103 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:52:51.693079 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b825e2cb-20f4-4e72-b52f-3e1857b3840a admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:51.700565 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24f30045-a82b-4e1a-8e01-a10821ddbf74 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:51.705756 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24f30045-a82b-4e1a-8e01-a10821ddbf74 service swift] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:51.706862 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b825e2cb-20f4-4e72-b52f-3e1857b3840a admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:51.712012 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5fdd40a-e7f1-4fab-80df-0d306af620f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:51.712893 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5fdd40a-e7f1-4fab-80df-0d306af620f4 admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:51.713628 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5fdd40a-e7f1-4fab-80df-0d306af620f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:51.714067 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5fdd40a-e7f1-4fab-80df-0d306af620f4 admin admin] PATH_INFO: `/v3/projects/23e083d3d1df4c00bb2a3c7d790cecc6/users/9eed8637742c4a9e901d6da821c4795e/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:51.729767 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 871/1745] 10.222.0.27 () {60 vars in 1384 bytes} [Sun Sep 11 00:52:51 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 631 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 11 00:52:51.739134 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5fdd40a-e7f1-4fab-80df-0d306af620f4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=23e083d3d1df4c00bb2a3c7d790cecc6, user_id=9eed8637742c4a9e901d6da821c4795e, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:51.742816 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5fdd40a-e7f1-4fab-80df-0d306af620f4 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:51.768438 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 875/1746] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:52:51 2022] PUT /identity/v3/projects/23e083d3d1df4c00bb2a3c7d790cecc6/users/9eed8637742c4a9e901d6da821c4795e/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 76 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:52:51.772044 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f30045-a82b-4e1a-8e01-a10821ddbf74 service swift] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:51.792690 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f30045-a82b-4e1a-8e01-a10821ddbf74 service swift] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:51.939909 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7cdcca-c2de-4d43-b9e5-7effaf537d68 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:51.939909 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7cdcca-c2de-4d43-b9e5-7effaf537d68 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:51.939909 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7cdcca-c2de-4d43-b9e5-7effaf537d68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:51.981403 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-921f4fa1-e7e0-4245-8eb6-9337823df460 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:51.981403 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921f4fa1-e7e0-4245-8eb6-9337823df460 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:51.981403 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921f4fa1-e7e0-4245-8eb6-9337823df460 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:51.981403 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921f4fa1-e7e0-4245-8eb6-9337823df460 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:51.981781 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-921f4fa1-e7e0-4245-8eb6-9337823df460 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:51.983554 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-921f4fa1-e7e0-4245-8eb6-9337823df460 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:52.050520 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 872/1747] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:52:51 2022] GET /identity/v3/roles => generated 1420 bytes in 276 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:52:52.051009 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-7c7cdcca-c2de-4d43-b9e5-7effaf537d68 None None] MFA Rules not processed for user `8b1fc2c958f045538fb15d7d1d4a408a`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:52.051009 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-7c7cdcca-c2de-4d43-b9e5-7effaf537d68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:52.052576 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-921f4fa1-e7e0-4245-8eb6-9337823df460 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:52.082533 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-921f4fa1-e7e0-4245-8eb6-9337823df460 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:52.111673 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c7dfb6-ca5d-4dbb-baff-df6243c7a718 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:52.119557 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c7dfb6-ca5d-4dbb-baff-df6243c7a718 admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:52.119557 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c7dfb6-ca5d-4dbb-baff-df6243c7a718 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:52.119557 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c7dfb6-ca5d-4dbb-baff-df6243c7a718 admin admin] PATH_INFO: `/v3/projects/23e083d3d1df4c00bb2a3c7d790cecc6/users/9eed8637742c4a9e901d6da821c4795e/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:52.131143 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45c7dfb6-ca5d-4dbb-baff-df6243c7a718 admin admin] RBAC: Authorizing `identity:create_grant(project_id=23e083d3d1df4c00bb2a3c7d790cecc6, user_id=9eed8637742c4a9e901d6da821c4795e, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:52.134000 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45c7dfb6-ca5d-4dbb-baff-df6243c7a718 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:52.188625 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 873/1748] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:52:52 2022] PUT /identity/v3/projects/23e083d3d1df4c00bb2a3c7d790cecc6/users/9eed8637742c4a9e901d6da821c4795e/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:52:52.190137 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247feb50-7214-4019-a091-f046fc204c3a None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:52.190514 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247feb50-7214-4019-a091-f046fc204c3a None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:52.192326 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247feb50-7214-4019-a091-f046fc204c3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:52.358252 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-247feb50-7214-4019-a091-f046fc204c3a None None] MFA Rules not processed for user `9eed8637742c4a9e901d6da821c4795e`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:52.370380 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-247feb50-7214-4019-a091-f046fc204c3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:52.486920 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 876/1749] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:51 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 562 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:52:52.494276 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8863e05b-d480-4171-ab0b-a552f0ac306e None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:52.494689 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8863e05b-d480-4171-ab0b-a552f0ac306e None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:52.495133 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8863e05b-d480-4171-ab0b-a552f0ac306e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:52.556804 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 874/1750] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:52 2022] POST /identity/v3/auth/tokens => generated 3347 bytes in 367 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:52.557810 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-247feb50-7214-4019-a091-f046fc204c3a None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:52.565576 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-247feb50-7214-4019-a091-f046fc204c3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:52.631194 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-8863e05b-d480-4171-ab0b-a552f0ac306e None None] MFA Rules not processed for user `8b1fc2c958f045538fb15d7d1d4a408a`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:52.635932 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:52:52.642884 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-8863e05b-d480-4171-ab0b-a552f0ac306e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:52.699746 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:52:52.806866 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 877/1751] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:52 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 316 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:52.808196 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8863e05b-d480-4171-ab0b-a552f0ac306e None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:52.812180 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-8863e05b-d480-4171-ab0b-a552f0ac306e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:52.892448 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-247feb50-7214-4019-a091-f046fc204c3a None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:52.911511 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da448cd-1edf-4181-a5e6-6941af9fb665 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:52.912803 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da448cd-1edf-4181-a5e6-6941af9fb665 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:52.913274 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da448cd-1edf-4181-a5e6-6941af9fb665 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:52.913776 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da448cd-1edf-4181-a5e6-6941af9fb665 admin admin] PATH_INFO: `/v3/projects/d3c0ead056414321832a11081ed450ce` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:52.915688 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8da448cd-1edf-4181-a5e6-6941af9fb665 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d3c0ead056414321832a11081ed450ce)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:52.922266 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8da448cd-1edf-4181-a5e6-6941af9fb665 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:53.083824 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8863e05b-d480-4171-ab0b-a552f0ac306e None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:53.109095 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-8da448cd-1edf-4181-a5e6-6941af9fb665 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd3c0ead056414321832a11081ed450ce', 'request_id': 'req-8da448cd-1edf-4181-a5e6-6941af9fb665'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:53.133551 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-8da448cd-1edf-4181-a5e6-6941af9fb665 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd3c0ead056414321832a11081ed450ce', 'request_id': 'req-8da448cd-1edf-4181-a5e6-6941af9fb665'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:53.161835 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-8da448cd-1edf-4181-a5e6-6941af9fb665 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd3c0ead056414321832a11081ed450ce', 'request_id': 'req-8da448cd-1edf-4181-a5e6-6941af9fb665'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:53.197721 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-8da448cd-1edf-4181-a5e6-6941af9fb665 admin admin] The token cache is being invalidate because project d3c0ead056414321832a11081ed450ce was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87924) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:52:53.198214 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-8da448cd-1edf-4181-a5e6-6941af9fb665 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:53.201692 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-8da448cd-1edf-4181-a5e6-6941af9fb665 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:53.204364 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 875/1752] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:52:52 2022] DELETE /identity/v3/projects/d3c0ead056414321832a11081ed450ce => generated 0 bytes in 647 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:52:53.224899 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da448cd-1edf-4181-a5e6-6941af9fb665 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:53.229797 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-8da448cd-1edf-4181-a5e6-6941af9fb665 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:53.274538 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-086bfab2-c301-4e73-8725-f2fef4b98d35 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:53.276986 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086bfab2-c301-4e73-8725-f2fef4b98d35 service cinder] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:53.277741 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086bfab2-c301-4e73-8725-f2fef4b98d35 service cinder] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:53.278389 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086bfab2-c301-4e73-8725-f2fef4b98d35 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:53.283169 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-086bfab2-c301-4e73-8725-f2fef4b98d35 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:53.619438 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da448cd-1edf-4181-a5e6-6941af9fb665 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:53.644415 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-086bfab2-c301-4e73-8725-f2fef4b98d35 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:53.645897 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-086bfab2-c301-4e73-8725-f2fef4b98d35 service cinder] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:53.655290 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a8a15c0-fed9-4692-9506-78ed1df26bbc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '835def0e6c6d4365a2767051509eb0d6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'ResellerAdmin', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:53.657266 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8a15c0-fed9-4692-9506-78ed1df26bbc service nova] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:53.657765 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8a15c0-fed9-4692-9506-78ed1df26bbc service nova] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:53.658235 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8a15c0-fed9-4692-9506-78ed1df26bbc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:53.667358 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-3a8a15c0-fed9-4692-9506-78ed1df26bbc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:53.707984 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 878/1753] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:52:52 2022] GET /identity/v3/auth/tokens => generated 3347 bytes in 900 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:52:53.709031 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-086bfab2-c301-4e73-8725-f2fef4b98d35 service cinder] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:53.713813 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-086bfab2-c301-4e73-8725-f2fef4b98d35 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:53.877203 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:52:53.929449 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:52:53.985899 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a8a15c0-fed9-4692-9506-78ed1df26bbc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:53.993909 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a8a15c0-fed9-4692-9506-78ed1df26bbc service nova] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:54.027811 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 876/1754] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:52:53 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 817 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:52:54.059341 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-086bfab2-c301-4e73-8725-f2fef4b98d35 service cinder] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:54.072158 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea3a961-412c-4dae-ae1e-ae8dcb9259dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:54.074868 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea3a961-412c-4dae-ae1e-ae8dcb9259dc admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:54.075239 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea3a961-412c-4dae-ae1e-ae8dcb9259dc admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:54.075526 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea3a961-412c-4dae-ae1e-ae8dcb9259dc admin admin] PATH_INFO: `/v3/users/1010109f1ef44b4eb7415b58c24741dc` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:54.078599 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dea3a961-412c-4dae-ae1e-ae8dcb9259dc admin admin] RBAC: Authorizing `identity:delete_user(user_id=1010109f1ef44b4eb7415b58c24741dc)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:54.085436 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dea3a961-412c-4dae-ae1e-ae8dcb9259dc admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:54.117894 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a8a15c0-fed9-4692-9506-78ed1df26bbc service nova] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:54.119386 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-3a8a15c0-fed9-4692-9506-78ed1df26bbc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:54.152559 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a8a15c0-fed9-4692-9506-78ed1df26bbc service nova] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:54.173914 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ebbf302-8c13-46e0-a09e-73cf0ce8ebcb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:54.175777 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ebbf302-8c13-46e0-a09e-73cf0ce8ebcb admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:54.176458 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ebbf302-8c13-46e0-a09e-73cf0ce8ebcb admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:54.177334 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ebbf302-8c13-46e0-a09e-73cf0ce8ebcb admin admin] PATH_INFO: `/v3/users/6b712be4c60a46e591301698da7f13dc` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:54.179396 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ebbf302-8c13-46e0-a09e-73cf0ce8ebcb admin admin] RBAC: Authorizing `identity:delete_user(user_id=6b712be4c60a46e591301698da7f13dc)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:54.190660 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ebbf302-8c13-46e0-a09e-73cf0ce8ebcb admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:54.216905 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-dea3a961-412c-4dae-ae1e-ae8dcb9259dc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1010109f1ef44b4eb7415b58c24741dc'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:54.218759 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-dea3a961-412c-4dae-ae1e-ae8dcb9259dc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1010109f1ef44b4eb7415b58c24741dc'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:54.219621 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-dea3a961-412c-4dae-ae1e-ae8dcb9259dc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1010109f1ef44b4eb7415b58c24741dc'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:54.236541 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-dea3a961-412c-4dae-ae1e-ae8dcb9259dc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1010109f1ef44b4eb7415b58c24741dc'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:54.284703 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-dea3a961-412c-4dae-ae1e-ae8dcb9259dc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1010109f1ef44b4eb7415b58c24741dc'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:54.320227 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-4ebbf302-8c13-46e0-a09e-73cf0ce8ebcb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6b712be4c60a46e591301698da7f13dc'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:54.322712 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-4ebbf302-8c13-46e0-a09e-73cf0ce8ebcb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6b712be4c60a46e591301698da7f13dc'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:54.324713 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-4ebbf302-8c13-46e0-a09e-73cf0ce8ebcb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6b712be4c60a46e591301698da7f13dc'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:54.328816 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 879/1755] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:52:53 2022] DELETE /identity/v3/users/1010109f1ef44b4eb7415b58c24741dc => generated 0 bytes in 620 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:52:54.349527 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-4ebbf302-8c13-46e0-a09e-73cf0ce8ebcb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6b712be4c60a46e591301698da7f13dc'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:54.385442 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-4ebbf302-8c13-46e0-a09e-73cf0ce8ebcb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6b712be4c60a46e591301698da7f13dc'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:54.415589 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 877/1756] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:52:54 2022] DELETE /identity/v3/users/6b712be4c60a46e591301698da7f13dc => generated 0 bytes in 298 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:52:54.421351 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37966150-c3c5-4fb2-a615-611c275160af None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:54.421676 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37966150-c3c5-4fb2-a615-611c275160af None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:54.421951 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37966150-c3c5-4fb2-a615-611c275160af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:54.474155 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-37966150-c3c5-4fb2-a615-611c275160af None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:54.477660 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-37966150-c3c5-4fb2-a615-611c275160af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:54.645026 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 880/1757] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 223 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:55.104532 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ebbf302-8c13-46e0-a09e-73cf0ce8ebcb admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:55.107202 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-4ebbf302-8c13-46e0-a09e-73cf0ce8ebcb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:55.548384 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ebbf302-8c13-46e0-a09e-73cf0ce8ebcb admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:55.573887 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9902405b-0156-4327-834d-2ffdd22041bc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:55.577453 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9902405b-0156-4327-834d-2ffdd22041bc service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:55.578630 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9902405b-0156-4327-834d-2ffdd22041bc service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:55.579422 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9902405b-0156-4327-834d-2ffdd22041bc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:55.600000 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9902405b-0156-4327-834d-2ffdd22041bc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:55.602923 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9902405b-0156-4327-834d-2ffdd22041bc service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:55.629335 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 878/1758] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:52:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 526 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:52:57.925872 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0950078-b4cf-4d33-938c-f1bc969ba41f None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:57.926603 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0950078-b4cf-4d33-938c-f1bc969ba41f None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:57.926603 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0950078-b4cf-4d33-938c-f1bc969ba41f None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:57.928294 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 881/1759] 10.222.0.27 () {54 vars in 912 bytes} [Sun Sep 11 00:52:57 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:52:57.958672 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb98a2c3-d396-4487-87d7-cebb3db9bf68 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:57.959193 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb98a2c3-d396-4487-87d7-cebb3db9bf68 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:57.959312 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb98a2c3-d396-4487-87d7-cebb3db9bf68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:58.047204 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-bb98a2c3-d396-4487-87d7-cebb3db9bf68 None None] MFA Rules not processed for user `a5f7b13283cb4ebfb25160aa6ad0e2f4`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:58.054992 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-bb98a2c3-d396-4487-87d7-cebb3db9bf68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:58.064982 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4868ad-b4f2-4c62-b9f4-b25b88fb1386 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:58.065385 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4868ad-b4f2-4c62-b9f4-b25b88fb1386 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:58.065821 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4868ad-b4f2-4c62-b9f4-b25b88fb1386 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:58.108898 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 879/1760] 10.222.0.27 () {58 vars in 1009 bytes} [Sun Sep 11 00:52:57 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:52:58.156720 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-3f4868ad-b4f2-4c62-b9f4-b25b88fb1386 None None] MFA Rules not processed for user `9048531ea3c64fe793aec2de9b56b2bf`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:58.158108 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-3f4868ad-b4f2-4c62-b9f4-b25b88fb1386 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:58.176229 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fd9678-3c9f-4df6-9438-1afd84a03ae2 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:58.176834 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fd9678-3c9f-4df6-9438-1afd84a03ae2 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:58.177250 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fd9678-3c9f-4df6-9438-1afd84a03ae2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:58.286951 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-49fd9678-3c9f-4df6-9438-1afd84a03ae2 None None] MFA Rules not processed for user `a5f7b13283cb4ebfb25160aa6ad0e2f4`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:58.294614 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-49fd9678-3c9f-4df6-9438-1afd84a03ae2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:58.333651 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 880/1761] 10.222.0.18 () {58 vars in 1015 bytes} [Sun Sep 11 00:52:58 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:58.339429 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49fd9678-3c9f-4df6-9438-1afd84a03ae2 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:58.345734 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-49fd9678-3c9f-4df6-9438-1afd84a03ae2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:58.395810 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49fd9678-3c9f-4df6-9438-1afd84a03ae2 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:58.410497 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be257cf-b6ea-4b67-92d7-a5df9c107551 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:58.411229 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:52:58.411679 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be257cf-b6ea-4b67-92d7-a5df9c107551 service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:58.412329 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be257cf-b6ea-4b67-92d7-a5df9c107551 service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:58.412329 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be257cf-b6ea-4b67-92d7-a5df9c107551 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:58.450023 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7be257cf-b6ea-4b67-92d7-a5df9c107551 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:58.453194 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7be257cf-b6ea-4b67-92d7-a5df9c107551 service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:58.494590 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 881/1762] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:52:58 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 160 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:52:58.499038 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 882/1763] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:58 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 432 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:58.500713 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be257cf-b6ea-4b67-92d7-a5df9c107551 service neutron] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:58.502500 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-7be257cf-b6ea-4b67-92d7-a5df9c107551 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:58.547774 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be257cf-b6ea-4b67-92d7-a5df9c107551 service neutron] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:58.562637 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da7eab80-832b-4281-997f-072b7c9d0cc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:58.563531 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da7eab80-832b-4281-997f-072b7c9d0cc8 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:58.563936 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da7eab80-832b-4281-997f-072b7c9d0cc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:58.564292 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da7eab80-832b-4281-997f-072b7c9d0cc8 admin admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:58.565019 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da7eab80-832b-4281-997f-072b7c9d0cc8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:58.567415 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da7eab80-832b-4281-997f-072b7c9d0cc8 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:58.620054 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 882/1764] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:52:58 2022] POST /identity/v3/projects => generated 371 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:52:58.621302 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f4868ad-b4f2-4c62-b9f4-b25b88fb1386 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:58.643316 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f4868ad-b4f2-4c62-b9f4-b25b88fb1386 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:58.820996 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28cf0a99-cc07-4377-88d1-d29fc8ca469b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:58.822289 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28cf0a99-cc07-4377-88d1-d29fc8ca469b admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:58.822760 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28cf0a99-cc07-4377-88d1-d29fc8ca469b admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:58.823266 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28cf0a99-cc07-4377-88d1-d29fc8ca469b admin admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:58.824333 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28cf0a99-cc07-4377-88d1-d29fc8ca469b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:58.827604 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28cf0a99-cc07-4377-88d1-d29fc8ca469b admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:58.840162 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da7eab80-832b-4281-997f-072b7c9d0cc8 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:58.851856 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:52:58.886818 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 883/1765] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:52:58 2022] POST /identity/v3/users => generated 292 bytes in 266 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:52:58.892926 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28cf0a99-cc07-4377-88d1-d29fc8ca469b admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:58.929240 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28cf0a99-cc07-4377-88d1-d29fc8ca469b admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:58.948834 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a4f61dd-55ad-42f1-a7ff-bb8dcf4a1638 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:58.951087 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a4f61dd-55ad-42f1-a7ff-bb8dcf4a1638 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:58.953390 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a4f61dd-55ad-42f1-a7ff-bb8dcf4a1638 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:58.953939 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a4f61dd-55ad-42f1-a7ff-bb8dcf4a1638 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:58.954864 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a4f61dd-55ad-42f1-a7ff-bb8dcf4a1638 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:58.957840 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a4f61dd-55ad-42f1-a7ff-bb8dcf4a1638 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:59.023954 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 884/1766] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:52:58 2022] GET /identity/v3/roles => generated 1420 bytes in 132 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:52:59.025730 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8920fdc1-cce4-402b-8270-2d6924a320af None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:59.026719 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8920fdc1-cce4-402b-8270-2d6924a320af None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:59.027331 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8920fdc1-cce4-402b-8270-2d6924a320af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:59.039061 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da7eab80-832b-4281-997f-072b7c9d0cc8 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:59.050928 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-274e6928-abdc-4da5-8477-78725e6edb63 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:59.051916 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-274e6928-abdc-4da5-8477-78725e6edb63 service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:59.052747 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-274e6928-abdc-4da5-8477-78725e6edb63 service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:59.054115 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-274e6928-abdc-4da5-8477-78725e6edb63 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:59.071817 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-274e6928-abdc-4da5-8477-78725e6edb63 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:59.075790 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-274e6928-abdc-4da5-8477-78725e6edb63 service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:59.085253 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-8920fdc1-cce4-402b-8270-2d6924a320af None None] MFA Rules not processed for user `73a3d3c48913474ab8fa8fedd1bc7d03`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:52:59.088379 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-8920fdc1-cce4-402b-8270-2d6924a320af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:59.099965 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 883/1767] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:52:58 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 260 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:52:59.101395 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-274e6928-abdc-4da5-8477-78725e6edb63 service neutron] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:59.129494 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-274e6928-abdc-4da5-8477-78725e6edb63 service neutron] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:59.143762 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad64553f-035f-4784-8204-3b14367e249b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:59.144837 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad64553f-035f-4784-8204-3b14367e249b admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:59.146959 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad64553f-035f-4784-8204-3b14367e249b admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:59.147728 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad64553f-035f-4784-8204-3b14367e249b admin admin] PATH_INFO: `/v3/projects/f8c5ffd6786c4a3f9bd9462db458af33/users/235c65cabb824f27b4ee5c49f6b6f878/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:59.199695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad64553f-035f-4784-8204-3b14367e249b admin admin] RBAC: Authorizing `identity:create_grant(project_id=f8c5ffd6786c4a3f9bd9462db458af33, user_id=235c65cabb824f27b4ee5c49f6b6f878, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:59.207157 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad64553f-035f-4784-8204-3b14367e249b admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:59.253663 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 884/1768] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:52:59 2022] PUT /identity/v3/projects/f8c5ffd6786c4a3f9bd9462db458af33/users/235c65cabb824f27b4ee5c49f6b6f878/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:52:59.255618 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad64553f-035f-4784-8204-3b14367e249b admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:59.262469 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 885/1769] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:59 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 237 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:52:59.379477 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad64553f-035f-4784-8204-3b14367e249b admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:59.387278 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa54d415-e955-41b3-b232-35da22f91030 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:59.388053 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa54d415-e955-41b3-b232-35da22f91030 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:59.388337 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa54d415-e955-41b3-b232-35da22f91030 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:59.388909 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa54d415-e955-41b3-b232-35da22f91030 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:59.391293 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa54d415-e955-41b3-b232-35da22f91030 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:59.394484 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa54d415-e955-41b3-b232-35da22f91030 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:59.419412 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 885/1770] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:52:59 2022] GET /identity/v3/roles => generated 1420 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:52:59.421465 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8920fdc1-cce4-402b-8270-2d6924a320af None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:59.463605 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa54d415-e955-41b3-b232-35da22f91030 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:59.474322 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-aa54d415-e955-41b3-b232-35da22f91030 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:59.486847 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8920fdc1-cce4-402b-8270-2d6924a320af None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:59.501262 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ced0cf-76c3-4116-b21d-baff6191dacd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:59.505788 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ced0cf-76c3-4116-b21d-baff6191dacd admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:59.505788 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ced0cf-76c3-4116-b21d-baff6191dacd admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:59.505788 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ced0cf-76c3-4116-b21d-baff6191dacd admin admin] PATH_INFO: `/v3/projects/f8c5ffd6786c4a3f9bd9462db458af33/users/235c65cabb824f27b4ee5c49f6b6f878/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:59.506299 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8ced0cf-76c3-4116-b21d-baff6191dacd admin admin] RBAC: Authorizing `identity:create_grant(project_id=f8c5ffd6786c4a3f9bd9462db458af33, user_id=235c65cabb824f27b4ee5c49f6b6f878, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:59.511806 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8ced0cf-76c3-4116-b21d-baff6191dacd admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:59.554232 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 886/1771] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:52:59 2022] PUT /identity/v3/projects/f8c5ffd6786c4a3f9bd9462db458af33/users/235c65cabb824f27b4ee5c49f6b6f878/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:52:59.557623 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa54d415-e955-41b3-b232-35da22f91030 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:59.558627 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ced0cf-76c3-4116-b21d-baff6191dacd admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:52:59.570457 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-e8ced0cf-76c3-4116-b21d-baff6191dacd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:52:59.726052 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-129c560c-9d98-443b-9e0c-c2825d5d3531 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:59.726719 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129c560c-9d98-443b-9e0c-c2825d5d3531 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:59.727032 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129c560c-9d98-443b-9e0c-c2825d5d3531 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:59.727309 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129c560c-9d98-443b-9e0c-c2825d5d3531 admin admin] PATH_INFO: `/v3/projects/0cf1b65c9a0545f7a009c34c8f8175c3` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:59.731231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-129c560c-9d98-443b-9e0c-c2825d5d3531 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0cf1b65c9a0545f7a009c34c8f8175c3)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:59.733267 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-129c560c-9d98-443b-9e0c-c2825d5d3531 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:59.789462 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:52:59.800948 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:52:59.806287 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ced0cf-76c3-4116-b21d-baff6191dacd admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:52:59.839660 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16d0959b-a10c-4477-841c-6b7261d72bcf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:52:59.840660 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d0959b-a10c-4477-841c-6b7261d72bcf admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:59.842933 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d0959b-a10c-4477-841c-6b7261d72bcf admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:59.848125 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d0959b-a10c-4477-841c-6b7261d72bcf admin admin] PATH_INFO: `/v3/projects/62dcebc4cf7b4f708957689493f34d55` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:52:59.853245 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16d0959b-a10c-4477-841c-6b7261d72bcf admin admin] RBAC: Authorizing `identity:delete_project(project_id=62dcebc4cf7b4f708957689493f34d55)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:52:59.860763 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16d0959b-a10c-4477-841c-6b7261d72bcf admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:52:59.877067 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-129c560c-9d98-443b-9e0c-c2825d5d3531 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0cf1b65c9a0545f7a009c34c8f8175c3', 'request_id': 'req-129c560c-9d98-443b-9e0c-c2825d5d3531'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:59.898942 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-129c560c-9d98-443b-9e0c-c2825d5d3531 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0cf1b65c9a0545f7a009c34c8f8175c3', 'request_id': 'req-129c560c-9d98-443b-9e0c-c2825d5d3531'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:59.916704 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-129c560c-9d98-443b-9e0c-c2825d5d3531 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0cf1b65c9a0545f7a009c34c8f8175c3', 'request_id': 'req-129c560c-9d98-443b-9e0c-c2825d5d3531'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:59.942603 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-129c560c-9d98-443b-9e0c-c2825d5d3531 admin admin] The token cache is being invalidate because project 0cf1b65c9a0545f7a009c34c8f8175c3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87924) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:52:59.947752 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-129c560c-9d98-443b-9e0c-c2825d5d3531 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:59.948587 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-129c560c-9d98-443b-9e0c-c2825d5d3531 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:52:59.954192 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 886/1772] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:52:59 2022] DELETE /identity/v3/projects/0cf1b65c9a0545f7a009c34c8f8175c3 => generated 0 bytes in 491 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:52:59.955648 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a00845-d4a2-4fe4-9159-cea2c7935d79 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:52:59.956109 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a00845-d4a2-4fe4-9159-cea2c7935d79 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:52:59.956396 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a00845-d4a2-4fe4-9159-cea2c7935d79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:00.054739 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-23a00845-d4a2-4fe4-9159-cea2c7935d79 None None] MFA Rules not processed for user `d886b168a8eb4424987555c073d71727`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:00.061207 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-23a00845-d4a2-4fe4-9159-cea2c7935d79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:00.101395 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-16d0959b-a10c-4477-841c-6b7261d72bcf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '62dcebc4cf7b4f708957689493f34d55', 'request_id': 'req-16d0959b-a10c-4477-841c-6b7261d72bcf'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:00.126508 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-16d0959b-a10c-4477-841c-6b7261d72bcf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '62dcebc4cf7b4f708957689493f34d55', 'request_id': 'req-16d0959b-a10c-4477-841c-6b7261d72bcf'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:00.148577 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-16d0959b-a10c-4477-841c-6b7261d72bcf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '62dcebc4cf7b4f708957689493f34d55', 'request_id': 'req-16d0959b-a10c-4477-841c-6b7261d72bcf'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:00.170717 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-16d0959b-a10c-4477-841c-6b7261d72bcf admin admin] The token cache is being invalidate because project 62dcebc4cf7b4f708957689493f34d55 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87923) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:53:00.171250 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-16d0959b-a10c-4477-841c-6b7261d72bcf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:00.172563 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-16d0959b-a10c-4477-841c-6b7261d72bcf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:00.178550 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 887/1773] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:52:59 2022] DELETE /identity/v3/projects/62dcebc4cf7b4f708957689493f34d55 => generated 0 bytes in 620 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:00.179774 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16d0959b-a10c-4477-841c-6b7261d72bcf admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:00.191249 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-16d0959b-a10c-4477-841c-6b7261d72bcf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:00.374998 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 887/1774] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:52:59 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 424 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:53:00.379527 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b88544-8099-4e00-b56e-8832599644d3 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:00.380972 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b88544-8099-4e00-b56e-8832599644d3 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:00.381359 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b88544-8099-4e00-b56e-8832599644d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:00.424644 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-a1b88544-8099-4e00-b56e-8832599644d3 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:00.429745 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-a1b88544-8099-4e00-b56e-8832599644d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:00.507232 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16d0959b-a10c-4477-841c-6b7261d72bcf admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:00.522227 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-220739f0-ccec-4c38-adea-e3d2bde282dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:00.523134 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220739f0-ccec-4c38-adea-e3d2bde282dc admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:00.523420 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220739f0-ccec-4c38-adea-e3d2bde282dc admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:00.523746 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220739f0-ccec-4c38-adea-e3d2bde282dc admin admin] PATH_INFO: `/v3/users/9eed8637742c4a9e901d6da821c4795e` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:00.525175 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-220739f0-ccec-4c38-adea-e3d2bde282dc admin admin] RBAC: Authorizing `identity:delete_user(user_id=9eed8637742c4a9e901d6da821c4795e)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:00.527635 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-220739f0-ccec-4c38-adea-e3d2bde282dc admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:00.643512 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-220739f0-ccec-4c38-adea-e3d2bde282dc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9eed8637742c4a9e901d6da821c4795e'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:00.645631 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-220739f0-ccec-4c38-adea-e3d2bde282dc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9eed8637742c4a9e901d6da821c4795e'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:00.648285 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-220739f0-ccec-4c38-adea-e3d2bde282dc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9eed8637742c4a9e901d6da821c4795e'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:00.672098 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-220739f0-ccec-4c38-adea-e3d2bde282dc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9eed8637742c4a9e901d6da821c4795e'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:00.708190 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 888/1775] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 323 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:53:00.710385 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e49810-7298-4bab-a8f0-29c55d0d4083 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:00.712701 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e49810-7298-4bab-a8f0-29c55d0d4083 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:00.713960 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e49810-7298-4bab-a8f0-29c55d0d4083 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:00.715713 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-220739f0-ccec-4c38-adea-e3d2bde282dc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9eed8637742c4a9e901d6da821c4795e'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:00.757592 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 888/1776] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:53:00 2022] DELETE /identity/v3/users/9eed8637742c4a9e901d6da821c4795e => generated 0 bytes in 578 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:00.766371 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-d2e49810-7298-4bab-a8f0-29c55d0d4083 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:00.767872 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-d2e49810-7298-4bab-a8f0-29c55d0d4083 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:00.870802 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:00.967831 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 889/1777] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 259 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:53:00.972242 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-220739f0-ccec-4c38-adea-e3d2bde282dc admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:00.982606 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:01.006548 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-220739f0-ccec-4c38-adea-e3d2bde282dc admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:01.039834 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68eb2f9b-4da4-4662-9a1b-ccdf54e4e4d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:01.040639 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68eb2f9b-4da4-4662-9a1b-ccdf54e4e4d4 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:01.041960 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68eb2f9b-4da4-4662-9a1b-ccdf54e4e4d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:01.059534 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68eb2f9b-4da4-4662-9a1b-ccdf54e4e4d4 admin admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:01.061714 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68eb2f9b-4da4-4662-9a1b-ccdf54e4e4d4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:53:01.062121 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68eb2f9b-4da4-4662-9a1b-ccdf54e4e4d4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:01.066832 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68eb2f9b-4da4-4662-9a1b-ccdf54e4e4d4 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:01.109085 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 889/1778] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 00:53:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:53:01.114808 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e49810-7298-4bab-a8f0-29c55d0d4083 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:01.157282 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e49810-7298-4bab-a8f0-29c55d0d4083 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:01.183170 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfab50f3-4034-419c-af24-4e50875ad9ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:01.183857 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfab50f3-4034-419c-af24-4e50875ad9ce admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:01.184131 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfab50f3-4034-419c-af24-4e50875ad9ce admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:01.184397 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfab50f3-4034-419c-af24-4e50875ad9ce admin admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:01.185053 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfab50f3-4034-419c-af24-4e50875ad9ce admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:01.192285 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68eb2f9b-4da4-4662-9a1b-ccdf54e4e4d4 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:01.194127 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-68eb2f9b-4da4-4662-9a1b-ccdf54e4e4d4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:01.204409 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfab50f3-4034-419c-af24-4e50875ad9ce admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:01.293154 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 890/1779] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:53:01 2022] POST /identity/v3/projects => generated 379 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:53:01.302649 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfab50f3-4034-419c-af24-4e50875ad9ce admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:01.425856 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68eb2f9b-4da4-4662-9a1b-ccdf54e4e4d4 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:01.573713 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfab50f3-4034-419c-af24-4e50875ad9ce admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:01.582597 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a55a4abf-f0a9-49ad-a98d-0672f883c0af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:01.585769 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a55a4abf-f0a9-49ad-a98d-0672f883c0af admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:01.586468 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a55a4abf-f0a9-49ad-a98d-0672f883c0af admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:01.586983 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a55a4abf-f0a9-49ad-a98d-0672f883c0af admin admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:01.587934 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a55a4abf-f0a9-49ad-a98d-0672f883c0af admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:01.592785 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a55a4abf-f0a9-49ad-a98d-0672f883c0af admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:01.692081 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adc6b813-3137-4522-b7db-c9fb1245532a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '835def0e6c6d4365a2767051509eb0d6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'ResellerAdmin', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:01.692870 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc6b813-3137-4522-b7db-c9fb1245532a service nova] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:01.693144 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc6b813-3137-4522-b7db-c9fb1245532a service nova] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:01.693422 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc6b813-3137-4522-b7db-c9fb1245532a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:01.698504 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 891/1780] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:53:01 2022] POST /identity/v3/users => generated 290 bytes in 394 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:53:01.707112 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a55a4abf-f0a9-49ad-a98d-0672f883c0af admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:01.711348 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-adc6b813-3137-4522-b7db-c9fb1245532a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:01.740903 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a55a4abf-f0a9-49ad-a98d-0672f883c0af admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:01.761689 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e411461-da1d-40e6-a32c-260c8700615d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:01.762327 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e411461-da1d-40e6-a32c-260c8700615d admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:01.762715 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e411461-da1d-40e6-a32c-260c8700615d admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:01.762891 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e411461-da1d-40e6-a32c-260c8700615d admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:01.763551 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e411461-da1d-40e6-a32c-260c8700615d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:01.765471 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e411461-da1d-40e6-a32c-260c8700615d admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:01.808837 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 892/1781] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:53:01 2022] GET /identity/v3/roles => generated 1420 bytes in 109 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:53:01.810325 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e411461-da1d-40e6-a32c-260c8700615d admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:01.833111 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e411461-da1d-40e6-a32c-260c8700615d admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:01.846803 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b249ec-7a5b-4166-98ef-dfbe9152af0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:01.847432 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b249ec-7a5b-4166-98ef-dfbe9152af0b admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:01.847733 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b249ec-7a5b-4166-98ef-dfbe9152af0b admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:01.848008 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b249ec-7a5b-4166-98ef-dfbe9152af0b admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:01.848692 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82b249ec-7a5b-4166-98ef-dfbe9152af0b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:01.850313 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82b249ec-7a5b-4166-98ef-dfbe9152af0b admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:01.884331 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 893/1782] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:53:01 2022] GET /identity/v3/roles => generated 1420 bytes in 74 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:53:01.890306 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b249ec-7a5b-4166-98ef-dfbe9152af0b admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:01.915769 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b249ec-7a5b-4166-98ef-dfbe9152af0b admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:01.927720 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a96b8409-9e15-4622-99d8-fdf21c51665b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:01.931017 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96b8409-9e15-4622-99d8-fdf21c51665b admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:01.932384 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96b8409-9e15-4622-99d8-fdf21c51665b admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:01.932987 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96b8409-9e15-4622-99d8-fdf21c51665b admin admin] PATH_INFO: `/v3/projects/94e07bd53dc44496a1bba2e64a36751e/users/92417d1a94d540b987e3b9856f753c38/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:01.968666 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adc6b813-3137-4522-b7db-c9fb1245532a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:01.970374 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adc6b813-3137-4522-b7db-c9fb1245532a service nova] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:01.982474 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a96b8409-9e15-4622-99d8-fdf21c51665b admin admin] RBAC: Authorizing `identity:create_grant(project_id=94e07bd53dc44496a1bba2e64a36751e, user_id=92417d1a94d540b987e3b9856f753c38, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:01.987266 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a96b8409-9e15-4622-99d8-fdf21c51665b admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:02.017673 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 890/1783] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:53:01 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 826 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:53:02.024036 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 894/1784] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:53:01 2022] PUT /identity/v3/projects/94e07bd53dc44496a1bba2e64a36751e/users/92417d1a94d540b987e3b9856f753c38/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:02.036608 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-532cdbfa-28f4-45ac-99c0-9ee36c23a22c None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:02.036931 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-532cdbfa-28f4-45ac-99c0-9ee36c23a22c None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:02.037171 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-532cdbfa-28f4-45ac-99c0-9ee36c23a22c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:02.207581 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-532cdbfa-28f4-45ac-99c0-9ee36c23a22c None None] MFA Rules not processed for user `92417d1a94d540b987e3b9856f753c38`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:02.215791 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-532cdbfa-28f4-45ac-99c0-9ee36c23a22c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:02.276521 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:02.475397 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 891/1785] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:02 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 440 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:53:02.475626 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a96b8409-9e15-4622-99d8-fdf21c51665b admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:02.760085 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a96b8409-9e15-4622-99d8-fdf21c51665b admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:02.774903 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483a480d-1e8f-48ed-99bb-6c36213b8a5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:02.775957 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483a480d-1e8f-48ed-99bb-6c36213b8a5f admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:02.782572 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483a480d-1e8f-48ed-99bb-6c36213b8a5f admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:02.784255 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483a480d-1e8f-48ed-99bb-6c36213b8a5f admin admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:02.786434 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-483a480d-1e8f-48ed-99bb-6c36213b8a5f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:02.796177 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-483a480d-1e8f-48ed-99bb-6c36213b8a5f admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:02.809704 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:02.875581 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 895/1786] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:53:02 2022] POST /identity/v3/projects => generated 377 bytes in 402 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:53:02.879935 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-532cdbfa-28f4-45ac-99c0-9ee36c23a22c None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:02.897163 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-532cdbfa-28f4-45ac-99c0-9ee36c23a22c None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:02.969209 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483a480d-1e8f-48ed-99bb-6c36213b8a5f admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:02.971469 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-483a480d-1e8f-48ed-99bb-6c36213b8a5f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:03.254458 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac0fb91-92eb-4f3c-9478-9bbebe2da7b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:03.258675 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac0fb91-92eb-4f3c-9478-9bbebe2da7b9 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:03.259429 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac0fb91-92eb-4f3c-9478-9bbebe2da7b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:03.260554 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac0fb91-92eb-4f3c-9478-9bbebe2da7b9 admin admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:03.262048 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac0fb91-92eb-4f3c-9478-9bbebe2da7b9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:03.269969 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac0fb91-92eb-4f3c-9478-9bbebe2da7b9 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:03.290508 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483a480d-1e8f-48ed-99bb-6c36213b8a5f admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:03.310600 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4f23988-7397-4f37-93c5-bfc4e920515f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:03.313352 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f23988-7397-4f37-93c5-bfc4e920515f admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:03.315755 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f23988-7397-4f37-93c5-bfc4e920515f admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:03.318712 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f23988-7397-4f37-93c5-bfc4e920515f admin admin] PATH_INFO: `/v3/projects/23e083d3d1df4c00bb2a3c7d790cecc6` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:03.325581 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4f23988-7397-4f37-93c5-bfc4e920515f admin admin] RBAC: Authorizing `identity:delete_project(project_id=23e083d3d1df4c00bb2a3c7d790cecc6)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:03.339177 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4f23988-7397-4f37-93c5-bfc4e920515f admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:03.355636 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 892/1787] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:53:02 2022] POST /identity/v3/users => generated 289 bytes in 476 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:53:03.359499 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac0fb91-92eb-4f3c-9478-9bbebe2da7b9 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:03.416237 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac0fb91-92eb-4f3c-9478-9bbebe2da7b9 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:03.456782 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dfc77d4-1cec-45b4-9700-b68c403c49de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:03.459511 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfc77d4-1cec-45b4-9700-b68c403c49de admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:03.459794 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfc77d4-1cec-45b4-9700-b68c403c49de admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:03.463416 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfc77d4-1cec-45b4-9700-b68c403c49de admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:03.465828 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dfc77d4-1cec-45b4-9700-b68c403c49de admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:03.467922 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dfc77d4-1cec-45b4-9700-b68c403c49de admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:03.512645 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-a4f23988-7397-4f37-93c5-bfc4e920515f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '23e083d3d1df4c00bb2a3c7d790cecc6', 'request_id': 'req-a4f23988-7397-4f37-93c5-bfc4e920515f'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:03.522926 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-a4f23988-7397-4f37-93c5-bfc4e920515f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '23e083d3d1df4c00bb2a3c7d790cecc6', 'request_id': 'req-a4f23988-7397-4f37-93c5-bfc4e920515f'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:03.528659 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 893/1788] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:53:03 2022] GET /identity/v3/roles => generated 1420 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:53:03.532978 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dfc77d4-1cec-45b4-9700-b68c403c49de admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:03.536603 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-a4f23988-7397-4f37-93c5-bfc4e920515f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '23e083d3d1df4c00bb2a3c7d790cecc6', 'request_id': 'req-a4f23988-7397-4f37-93c5-bfc4e920515f'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:03.580822 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dfc77d4-1cec-45b4-9700-b68c403c49de admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:03.586505 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-a4f23988-7397-4f37-93c5-bfc4e920515f admin admin] The token cache is being invalidate because project 23e083d3d1df4c00bb2a3c7d790cecc6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87924) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:53:03.586870 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-a4f23988-7397-4f37-93c5-bfc4e920515f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:03.590173 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-a4f23988-7397-4f37-93c5-bfc4e920515f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:03.600129 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 896/1789] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:53:02 2022] DELETE /identity/v3/projects/23e083d3d1df4c00bb2a3c7d790cecc6 => generated 0 bytes in 628 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:03.601940 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6fbbe56-9401-41ce-ba17-aa3ae54141d6 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:03.602227 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6fbbe56-9401-41ce-ba17-aa3ae54141d6 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:03.602508 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6fbbe56-9401-41ce-ba17-aa3ae54141d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:03.632987 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-c6fbbe56-9401-41ce-ba17-aa3ae54141d6 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:03.634971 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-c6fbbe56-9401-41ce-ba17-aa3ae54141d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:03.831219 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 897/1790] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 224 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:53:03.834997 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a00520-ad96-4d48-ada0-d111522fc5ee None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:03.834997 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a00520-ad96-4d48-ada0-d111522fc5ee None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:03.835281 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a00520-ad96-4d48-ada0-d111522fc5ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:03.844818 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9276d80c-8953-4228-98b2-085c7917acde admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:03.847011 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9276d80c-8953-4228-98b2-085c7917acde admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:03.849041 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9276d80c-8953-4228-98b2-085c7917acde admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:03.849559 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9276d80c-8953-4228-98b2-085c7917acde admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:03.850486 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9276d80c-8953-4228-98b2-085c7917acde admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:03.854909 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9276d80c-8953-4228-98b2-085c7917acde admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:03.886989 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 894/1791] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:53:03 2022] GET /identity/v3/roles => generated 1420 bytes in 355 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:53:03.887593 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-19a00520-ad96-4d48-ada0-d111522fc5ee None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:03.891301 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9276d80c-8953-4228-98b2-085c7917acde admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:03.891837 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-19a00520-ad96-4d48-ada0-d111522fc5ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:03.895723 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-9276d80c-8953-4228-98b2-085c7917acde admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:03.915917 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 898/1792] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:53:03.919659 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a00520-ad96-4d48-ada0-d111522fc5ee None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:03.934876 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9276d80c-8953-4228-98b2-085c7917acde admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:03.942319 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a00520-ad96-4d48-ada0-d111522fc5ee None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:03.950204 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe63db70-7bba-4349-a605-056e97d9572b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:03.953199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe63db70-7bba-4349-a605-056e97d9572b admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:03.953789 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe63db70-7bba-4349-a605-056e97d9572b admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:03.954266 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe63db70-7bba-4349-a605-056e97d9572b admin admin] PATH_INFO: `/v3/projects/fb3a1ee0776c43f4bc17b8b02264fc57/users/48a1c0c4c3da47e5a13582a2e167e03b/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:03.955896 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2770975a-4eb7-4532-9db4-ff9369279cec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:03.957338 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2770975a-4eb7-4532-9db4-ff9369279cec admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:03.958683 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2770975a-4eb7-4532-9db4-ff9369279cec admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:03.959296 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2770975a-4eb7-4532-9db4-ff9369279cec admin admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:03.961046 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2770975a-4eb7-4532-9db4-ff9369279cec admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:53:03.961693 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2770975a-4eb7-4532-9db4-ff9369279cec admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:03.967930 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2770975a-4eb7-4532-9db4-ff9369279cec admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:03.978119 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:04.001642 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe63db70-7bba-4349-a605-056e97d9572b admin admin] RBAC: Authorizing `identity:create_grant(project_id=fb3a1ee0776c43f4bc17b8b02264fc57, user_id=48a1c0c4c3da47e5a13582a2e167e03b, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:04.008149 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe63db70-7bba-4349-a605-056e97d9572b admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:04.043174 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 899/1793] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 00:53:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 124 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:53:04.048357 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2770975a-4eb7-4532-9db4-ff9369279cec admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:04.068979 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 895/1794] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:53:03 2022] PUT /identity/v3/projects/fb3a1ee0776c43f4bc17b8b02264fc57/users/48a1c0c4c3da47e5a13582a2e167e03b/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:04.080191 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2770975a-4eb7-4532-9db4-ff9369279cec admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:04.085392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17394757-5dfd-4b2d-90d1-5e989782a414 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:04.085674 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17394757-5dfd-4b2d-90d1-5e989782a414 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:04.085942 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17394757-5dfd-4b2d-90d1-5e989782a414 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:04.131943 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:04.204452 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-17394757-5dfd-4b2d-90d1-5e989782a414 None None] MFA Rules not processed for user `48a1c0c4c3da47e5a13582a2e167e03b`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:04.208679 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-17394757-5dfd-4b2d-90d1-5e989782a414 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:04.257072 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50b5bc8d-341a-4708-beca-3fa28127a3c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:04.258149 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b5bc8d-341a-4708-beca-3fa28127a3c2 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:04.258536 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b5bc8d-341a-4708-beca-3fa28127a3c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:04.258895 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b5bc8d-341a-4708-beca-3fa28127a3c2 admin admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:04.260735 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50b5bc8d-341a-4708-beca-3fa28127a3c2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:04.263117 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50b5bc8d-341a-4708-beca-3fa28127a3c2 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:04.364771 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 900/1795] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:53:04 2022] POST /identity/v3/projects => generated 403 bytes in 317 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:53:04.379363 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50b5bc8d-341a-4708-beca-3fa28127a3c2 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:04.478802 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 896/1796] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:04 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 394 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:53:04.484112 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17394757-5dfd-4b2d-90d1-5e989782a414 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:04.505189 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17394757-5dfd-4b2d-90d1-5e989782a414 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:04.635758 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50b5bc8d-341a-4708-beca-3fa28127a3c2 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:04.664245 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ed7a94b-fb43-4b17-b915-f46aef14dc6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:04.665440 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed7a94b-fb43-4b17-b915-f46aef14dc6b admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:04.665942 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed7a94b-fb43-4b17-b915-f46aef14dc6b admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:04.666564 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed7a94b-fb43-4b17-b915-f46aef14dc6b admin admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:04.668348 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48973418-05d0-4dc7-b209-7aef45e568db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:04.668524 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48973418-05d0-4dc7-b209-7aef45e568db admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:04.668754 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48973418-05d0-4dc7-b209-7aef45e568db admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:04.669292 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48973418-05d0-4dc7-b209-7aef45e568db admin admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:04.671670 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ed7a94b-fb43-4b17-b915-f46aef14dc6b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:04.671807 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48973418-05d0-4dc7-b209-7aef45e568db admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:04.673912 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48973418-05d0-4dc7-b209-7aef45e568db admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:04.678601 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ed7a94b-fb43-4b17-b915-f46aef14dc6b admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:04.782114 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 901/1797] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:53:04 2022] POST /identity/v3/users => generated 302 bytes in 404 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:53:04.787054 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48973418-05d0-4dc7-b209-7aef45e568db admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:04.796094 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 897/1798] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:53:04 2022] POST /identity/v3/projects => generated 379 bytes in 310 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:53:04.798979 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ed7a94b-fb43-4b17-b915-f46aef14dc6b admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:04.831848 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48973418-05d0-4dc7-b209-7aef45e568db admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:04.949699 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ed7a94b-fb43-4b17-b915-f46aef14dc6b admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:04.958698 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b52152-395a-4ac8-82b6-d8bbcf57bf2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:04.959858 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b52152-395a-4ac8-82b6-d8bbcf57bf2a admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:04.960416 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b52152-395a-4ac8-82b6-d8bbcf57bf2a admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:04.960941 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b52152-395a-4ac8-82b6-d8bbcf57bf2a admin admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:04.962093 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b52152-395a-4ac8-82b6-d8bbcf57bf2a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:04.966289 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b52152-395a-4ac8-82b6-d8bbcf57bf2a admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:04.997170 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5975766-7684-48f3-83e6-84b15d57f4c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:05.000481 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5975766-7684-48f3-83e6-84b15d57f4c8 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:05.000713 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5975766-7684-48f3-83e6-84b15d57f4c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:05.001107 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5975766-7684-48f3-83e6-84b15d57f4c8 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:05.001853 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5975766-7684-48f3-83e6-84b15d57f4c8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:05.005864 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5975766-7684-48f3-83e6-84b15d57f4c8 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:05.014435 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 898/1799] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:53:04 2022] POST /identity/v3/users => generated 296 bytes in 217 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:53:05.015601 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b52152-395a-4ac8-82b6-d8bbcf57bf2a admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:05.036047 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 902/1800] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:53:04 2022] GET /identity/v3/roles => generated 1420 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:53:05.040420 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5975766-7684-48f3-83e6-84b15d57f4c8 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:05.059779 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5975766-7684-48f3-83e6-84b15d57f4c8 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:05.071579 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b52152-395a-4ac8-82b6-d8bbcf57bf2a admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:05.084067 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-099259b3-198e-481b-91d1-cddbcfaa53c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:05.084824 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-099259b3-198e-481b-91d1-cddbcfaa53c5 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:05.085673 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-099259b3-198e-481b-91d1-cddbcfaa53c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:05.085673 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-099259b3-198e-481b-91d1-cddbcfaa53c5 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:05.086190 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-099259b3-198e-481b-91d1-cddbcfaa53c5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:05.090227 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-099259b3-198e-481b-91d1-cddbcfaa53c5 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:05.094110 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-378f753f-cb5e-404c-b7e0-ba4fa4dc3c46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:05.094460 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:05.094979 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378f753f-cb5e-404c-b7e0-ba4fa4dc3c46 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:05.095261 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378f753f-cb5e-404c-b7e0-ba4fa4dc3c46 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:05.095549 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378f753f-cb5e-404c-b7e0-ba4fa4dc3c46 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:05.096173 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-378f753f-cb5e-404c-b7e0-ba4fa4dc3c46 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:05.105505 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-378f753f-cb5e-404c-b7e0-ba4fa4dc3c46 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:05.136879 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:05.147764 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 903/1801] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:53:05 2022] GET /identity/v3/roles => generated 1420 bytes in 107 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 11 00:53:05.150756 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-099259b3-198e-481b-91d1-cddbcfaa53c5 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:05.156890 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 899/1802] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:53:05 2022] GET /identity/v3/roles => generated 1420 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:53:05.157923 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-378f753f-cb5e-404c-b7e0-ba4fa4dc3c46 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:05.174651 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-378f753f-cb5e-404c-b7e0-ba4fa4dc3c46 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:05.181188 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e40af59e-c0e9-46ff-abc4-1d93bb101716 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:05.181999 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e40af59e-c0e9-46ff-abc4-1d93bb101716 admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:05.182275 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e40af59e-c0e9-46ff-abc4-1d93bb101716 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:05.182576 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e40af59e-c0e9-46ff-abc4-1d93bb101716 admin admin] PATH_INFO: `/v3/projects/c124e43066d047a1870389d5e67814d1/users/6a9d7463d5cc4622b17809b3da23c5db/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:05.212533 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-099259b3-198e-481b-91d1-cddbcfaa53c5 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:05.226982 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e40af59e-c0e9-46ff-abc4-1d93bb101716 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c124e43066d047a1870389d5e67814d1, user_id=6a9d7463d5cc4622b17809b3da23c5db, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:05.230394 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff82208f-35ec-4692-9ef0-5c03b4cb42d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:05.232138 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff82208f-35ec-4692-9ef0-5c03b4cb42d8 admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:05.233671 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff82208f-35ec-4692-9ef0-5c03b4cb42d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:05.234374 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff82208f-35ec-4692-9ef0-5c03b4cb42d8 admin admin] PATH_INFO: `/v3/projects/9bcc6857806944d9a22e0f4d53674801/users/de25b2dc88d642789944b637157c078c/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:05.235566 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e40af59e-c0e9-46ff-abc4-1d93bb101716 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:05.276089 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff82208f-35ec-4692-9ef0-5c03b4cb42d8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9bcc6857806944d9a22e0f4d53674801, user_id=de25b2dc88d642789944b637157c078c, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:05.284631 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff82208f-35ec-4692-9ef0-5c03b4cb42d8 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:05.320056 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 904/1803] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:53:05 2022] PUT /identity/v3/projects/9bcc6857806944d9a22e0f4d53674801/users/de25b2dc88d642789944b637157c078c/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:05.325429 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 900/1804] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:53:05 2022] PUT /identity/v3/projects/c124e43066d047a1870389d5e67814d1/users/6a9d7463d5cc4622b17809b3da23c5db/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:05.325774 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb14454-e492-4041-b052-acc677f1d7c2 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:05.326317 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb14454-e492-4041-b052-acc677f1d7c2 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:05.331101 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb14454-e492-4041-b052-acc677f1d7c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:05.332669 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e40af59e-c0e9-46ff-abc4-1d93bb101716 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:05.378767 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-1cb14454-e492-4041-b052-acc677f1d7c2 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:05.381577 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-1cb14454-e492-4041-b052-acc677f1d7c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:05.614996 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 905/1805] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 294 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:53:05.646170 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963f946e-d62b-471d-8a71-5a0841a36860 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:05.646851 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963f946e-d62b-471d-8a71-5a0841a36860 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:05.646851 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963f946e-d62b-471d-8a71-5a0841a36860 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:05.661973 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e40af59e-c0e9-46ff-abc4-1d93bb101716 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:05.675661 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-431d2899-e748-4e32-9e70-8d9459b1ff6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:05.676288 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-431d2899-e748-4e32-9e70-8d9459b1ff6c admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:05.676556 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-431d2899-e748-4e32-9e70-8d9459b1ff6c admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:05.681192 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-431d2899-e748-4e32-9e70-8d9459b1ff6c admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:05.682048 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-431d2899-e748-4e32-9e70-8d9459b1ff6c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:05.689782 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-431d2899-e748-4e32-9e70-8d9459b1ff6c admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:05.767721 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 901/1806] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:53:05 2022] GET /identity/v3/roles => generated 1420 bytes in 436 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:53:05.774950 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-431d2899-e748-4e32-9e70-8d9459b1ff6c admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:05.785793 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-431d2899-e748-4e32-9e70-8d9459b1ff6c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:05.811716 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-963f946e-d62b-471d-8a71-5a0841a36860 None None] MFA Rules not processed for user `235c65cabb824f27b4ee5c49f6b6f878`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:05.819395 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-963f946e-d62b-471d-8a71-5a0841a36860 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:05.969257 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-431d2899-e748-4e32-9e70-8d9459b1ff6c admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:05.982250 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3828d856-c494-4625-98eb-19631dd53263 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:05.984802 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3828d856-c494-4625-98eb-19631dd53263 service neutron] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:05.986611 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3828d856-c494-4625-98eb-19631dd53263 service neutron] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:05.987705 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3828d856-c494-4625-98eb-19631dd53263 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:06.023059 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3828d856-c494-4625-98eb-19631dd53263 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:06.027520 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3828d856-c494-4625-98eb-19631dd53263 service neutron] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:06.059373 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 902/1807] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:53:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 291 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:53:06.062190 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3828d856-c494-4625-98eb-19631dd53263 service neutron] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:06.092085 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 906/1808] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:05 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 447 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:53:06.096617 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c49090-2945-484c-a6db-f5cba455516b None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:06.096920 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c49090-2945-484c-a6db-f5cba455516b None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:06.097232 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c49090-2945-484c-a6db-f5cba455516b None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:06.106320 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 907/1809] 10.222.0.27 () {54 vars in 921 bytes} [Sun Sep 11 00:53:06 2022] GET /identity => generated 270 bytes in 14 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:53:06.106471 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3828d856-c494-4625-98eb-19631dd53263 service neutron] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:06.113273 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c49090-2945-484c-a6db-f5cba455516b None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:06.117286 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-74c49090-2945-484c-a6db-f5cba455516b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:06.130574 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-582c69ac-9147-49e9-b903-5801c3177b95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:06.133301 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582c69ac-9147-49e9-b903-5801c3177b95 admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:06.137618 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582c69ac-9147-49e9-b903-5801c3177b95 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:06.138056 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582c69ac-9147-49e9-b903-5801c3177b95 admin admin] PATH_INFO: `/v3/projects/c124e43066d047a1870389d5e67814d1/users/6a9d7463d5cc4622b17809b3da23c5db/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:06.147363 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-582c69ac-9147-49e9-b903-5801c3177b95 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c124e43066d047a1870389d5e67814d1, user_id=6a9d7463d5cc4622b17809b3da23c5db, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:06.156651 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-582c69ac-9147-49e9-b903-5801c3177b95 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:06.222477 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 903/1810] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:53:06 2022] PUT /identity/v3/projects/c124e43066d047a1870389d5e67814d1/users/6a9d7463d5cc4622b17809b3da23c5db/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:06.226263 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cbadfd1-8cc3-4f39-9050-5f968950c3c4 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:06.226592 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cbadfd1-8cc3-4f39-9050-5f968950c3c4 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:06.231316 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cbadfd1-8cc3-4f39-9050-5f968950c3c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:06.356215 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c49090-2945-484c-a6db-f5cba455516b None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:06.374638 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-8cbadfd1-8cc3-4f39-9050-5f968950c3c4 None None] MFA Rules not processed for user `6a9d7463d5cc4622b17809b3da23c5db`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:06.375956 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-8cbadfd1-8cc3-4f39-9050-5f968950c3c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:06.399628 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:06.586078 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b368f175-74f8-418d-ae81-64dd19f8da26 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:06.586723 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b368f175-74f8-418d-ae81-64dd19f8da26 service cinder] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:06.587051 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b368f175-74f8-418d-ae81-64dd19f8da26 service cinder] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:06.594463 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b368f175-74f8-418d-ae81-64dd19f8da26 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:06.599912 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-b368f175-74f8-418d-ae81-64dd19f8da26 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:06.603134 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:06.670415 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 904/1811] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:06 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 446 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:53:06.671526 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff75230e-4183-41ca-b558-2b0cfd64d5d9 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:06.671835 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff75230e-4183-41ca-b558-2b0cfd64d5d9 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:06.674585 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff75230e-4183-41ca-b558-2b0cfd64d5d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:06.758139 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-ff75230e-4183-41ca-b558-2b0cfd64d5d9 None None] MFA Rules not processed for user `835def0e6c6d4365a2767051509eb0d6`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:06.764066 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-ff75230e-4183-41ca-b558-2b0cfd64d5d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:06.817141 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b368f175-74f8-418d-ae81-64dd19f8da26 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:06.819396 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b368f175-74f8-418d-ae81-64dd19f8da26 service cinder] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:06.861334 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 908/1812] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:53:06 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 747 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:53:06.861334 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b368f175-74f8-418d-ae81-64dd19f8da26 service cinder] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:06.869677 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-b368f175-74f8-418d-ae81-64dd19f8da26 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:06.907748 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b368f175-74f8-418d-ae81-64dd19f8da26 service cinder] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:06.919319 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe15606c-523b-4dba-8721-8df54e1f97f7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:06.921873 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe15606c-523b-4dba-8721-8df54e1f97f7 service cinder] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:06.922084 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe15606c-523b-4dba-8721-8df54e1f97f7 service cinder] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:06.922359 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe15606c-523b-4dba-8721-8df54e1f97f7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:06.951129 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe15606c-523b-4dba-8721-8df54e1f97f7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:06.953394 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe15606c-523b-4dba-8721-8df54e1f97f7 service cinder] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:06.978271 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 905/1813] 10.222.0.27 () {58 vars in 1018 bytes} [Sun Sep 11 00:53:06 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 308 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:53:06.984192 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 909/1814] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:53:06 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:53:07.094168 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4be4aa0-01c6-419a-9f9b-f175c17c28b8 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:07.094401 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4be4aa0-01c6-419a-9f9b-f175c17c28b8 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:07.094676 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4be4aa0-01c6-419a-9f9b-f175c17c28b8 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:07.096084 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 906/1815] 10.222.0.27 () {54 vars in 921 bytes} [Sun Sep 11 00:53:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:53:07.136946 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ddc4e67-fac7-4223-8ea3-50ab959eec7f None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:07.138179 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ddc4e67-fac7-4223-8ea3-50ab959eec7f None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:07.138652 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ddc4e67-fac7-4223-8ea3-50ab959eec7f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:07.199844 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4be4aa0-01c6-419a-9f9b-f175c17c28b8 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:07.202031 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-c4be4aa0-01c6-419a-9f9b-f175c17c28b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:07.217697 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-4ddc4e67-fac7-4223-8ea3-50ab959eec7f None None] MFA Rules not processed for user `835def0e6c6d4365a2767051509eb0d6`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:07.219180 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-4ddc4e67-fac7-4223-8ea3-50ab959eec7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:07.237412 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4be4aa0-01c6-419a-9f9b-f175c17c28b8 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:07.249737 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 910/1816] 10.222.0.27 () {58 vars in 1018 bytes} [Sun Sep 11 00:53:07 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:53:07.250606 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ddc4e67-fac7-4223-8ea3-50ab959eec7f None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:07.251298 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd9f8940-d3a4-4ef3-af43-d9392eb15518 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '835def0e6c6d4365a2767051509eb0d6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'ResellerAdmin', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:07.252419 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9f8940-d3a4-4ef3-af43-d9392eb15518 service nova] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:07.252707 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9f8940-d3a4-4ef3-af43-d9392eb15518 service nova] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:07.252974 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9f8940-d3a4-4ef3-af43-d9392eb15518 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:07.267782 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ddc4e67-fac7-4223-8ea3-50ab959eec7f None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:07.276106 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd9f8940-d3a4-4ef3-af43-d9392eb15518 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:07.276582 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe6bbb0-57cc-4808-8449-b16fc1c3f117 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:07.277319 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe6bbb0-57cc-4808-8449-b16fc1c3f117 service cinder] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:07.278420 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe6bbb0-57cc-4808-8449-b16fc1c3f117 service cinder] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:07.278701 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe6bbb0-57cc-4808-8449-b16fc1c3f117 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:07.281075 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd9f8940-d3a4-4ef3-af43-d9392eb15518 service nova] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:07.284248 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-ebe6bbb0-57cc-4808-8449-b16fc1c3f117 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:07.312582 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 907/1817] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:53:07 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:53:07.321083 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd9f8940-d3a4-4ef3-af43-d9392eb15518 service nova] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:07.322672 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-dd9f8940-d3a4-4ef3-af43-d9392eb15518 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:07.362386 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd9f8940-d3a4-4ef3-af43-d9392eb15518 service nova] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:07.380284 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d55a90f6-85b6-4638-bb7a-7cc3a1417385 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '835def0e6c6d4365a2767051509eb0d6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'ResellerAdmin', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:07.381235 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55a90f6-85b6-4638-bb7a-7cc3a1417385 service nova] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:07.381381 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55a90f6-85b6-4638-bb7a-7cc3a1417385 service nova] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:07.381552 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55a90f6-85b6-4638-bb7a-7cc3a1417385 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:07.404561 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d55a90f6-85b6-4638-bb7a-7cc3a1417385 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:07.410964 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d55a90f6-85b6-4638-bb7a-7cc3a1417385 service nova] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:07.446613 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 908/1818] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:53:07 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 11 00:53:07.455597 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebe6bbb0-57cc-4808-8449-b16fc1c3f117 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:07.457787 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebe6bbb0-57cc-4808-8449-b16fc1c3f117 service cinder] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:07.494181 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 911/1819] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:53:07 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 244 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:53:08.870657 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d55a90f6-85b6-4638-bb7a-7cc3a1417385 service nova] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:08.896514 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d55a90f6-85b6-4638-bb7a-7cc3a1417385 service nova] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:08.910352 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1efd93a-405a-430a-bb61-c743c3e554fc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:08.911298 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1efd93a-405a-430a-bb61-c743c3e554fc service cinder] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:08.911762 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1efd93a-405a-430a-bb61-c743c3e554fc service cinder] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:08.912033 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1efd93a-405a-430a-bb61-c743c3e554fc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:09.019515 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1efd93a-405a-430a-bb61-c743c3e554fc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:09.026424 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1efd93a-405a-430a-bb61-c743c3e554fc service cinder] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:09.047407 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 909/1820] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:53:08 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 177 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:53:09.736384 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac20777-b657-4886-aaad-3bb1d9453862 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:09.737030 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac20777-b657-4886-aaad-3bb1d9453862 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:09.737030 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac20777-b657-4886-aaad-3bb1d9453862 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:09.813506 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-dac20777-b657-4886-aaad-3bb1d9453862 None None] MFA Rules not processed for user `de25b2dc88d642789944b637157c078c`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:09.815220 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-dac20777-b657-4886-aaad-3bb1d9453862 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:09.929110 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:10.001687 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 912/1821] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:09 2022] POST /identity/v3/auth/tokens => generated 3280 bytes in 266 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:53:10.005960 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1efd93a-405a-430a-bb61-c743c3e554fc service cinder] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:10.132485 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:10.168355 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1efd93a-405a-430a-bb61-c743c3e554fc service cinder] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:10.181093 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-748eaa8b-72ae-43df-842c-7923dc474174 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:10.181939 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748eaa8b-72ae-43df-842c-7923dc474174 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:10.182357 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748eaa8b-72ae-43df-842c-7923dc474174 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:10.182941 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748eaa8b-72ae-43df-842c-7923dc474174 admin admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:10.183913 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-748eaa8b-72ae-43df-842c-7923dc474174 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:10.186895 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-748eaa8b-72ae-43df-842c-7923dc474174 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:10.251509 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 910/1822] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:53:10 2022] POST /identity/v3/projects => generated 403 bytes in 246 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:53:10.255641 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac20777-b657-4886-aaad-3bb1d9453862 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:10.273877 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac20777-b657-4886-aaad-3bb1d9453862 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:10.412856 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86008f2c-3c50-40c3-9268-60a562ce3a90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:10.415180 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86008f2c-3c50-40c3-9268-60a562ce3a90 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:10.415450 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86008f2c-3c50-40c3-9268-60a562ce3a90 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:10.415728 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86008f2c-3c50-40c3-9268-60a562ce3a90 admin admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:10.416420 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86008f2c-3c50-40c3-9268-60a562ce3a90 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:10.422436 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86008f2c-3c50-40c3-9268-60a562ce3a90 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:10.481403 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 913/1823] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:53:10 2022] POST /identity/v3/users => generated 308 bytes in 225 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:53:10.484587 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-748eaa8b-72ae-43df-842c-7923dc474174 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:10.497259 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-748eaa8b-72ae-43df-842c-7923dc474174 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:10.510978 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4736c979-3842-45a9-bfaa-f5b485b6cf31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:10.511817 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4736c979-3842-45a9-bfaa-f5b485b6cf31 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:10.512243 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4736c979-3842-45a9-bfaa-f5b485b6cf31 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:10.512369 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3e8632-d5f9-40e0-8b47-184ce6fd7c93 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:10.512626 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3e8632-d5f9-40e0-8b47-184ce6fd7c93 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:10.512901 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3e8632-d5f9-40e0-8b47-184ce6fd7c93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:10.513056 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4736c979-3842-45a9-bfaa-f5b485b6cf31 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:10.513904 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4736c979-3842-45a9-bfaa-f5b485b6cf31 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:10.515716 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4736c979-3842-45a9-bfaa-f5b485b6cf31 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:10.528629 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 911/1824] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:53:10 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:53:10.533477 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4736c979-3842-45a9-bfaa-f5b485b6cf31 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:10.562396 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4736c979-3842-45a9-bfaa-f5b485b6cf31 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:10.569178 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-516b1e2f-670f-4591-ba3f-6c412bdd24ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:10.570269 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516b1e2f-670f-4591-ba3f-6c412bdd24ff admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:10.570692 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516b1e2f-670f-4591-ba3f-6c412bdd24ff admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:10.571087 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-1b3e8632-d5f9-40e0-8b47-184ce6fd7c93 None None] MFA Rules not processed for user `d886b168a8eb4424987555c073d71727`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:10.571309 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516b1e2f-670f-4591-ba3f-6c412bdd24ff admin admin] PATH_INFO: `/v3/projects/2902b795d287496fbf3688d0feedbc47/users/d2976f8d35684109a2fc6e93cbc547fa/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:10.572361 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-1b3e8632-d5f9-40e0-8b47-184ce6fd7c93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:10.588827 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-516b1e2f-670f-4591-ba3f-6c412bdd24ff admin admin] RBAC: Authorizing `identity:create_grant(project_id=2902b795d287496fbf3688d0feedbc47, user_id=d2976f8d35684109a2fc6e93cbc547fa, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:10.592840 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-516b1e2f-670f-4591-ba3f-6c412bdd24ff admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:10.619886 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 912/1825] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:53:10 2022] PUT /identity/v3/projects/2902b795d287496fbf3688d0feedbc47/users/d2976f8d35684109a2fc6e93cbc547fa/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 87 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:10.623533 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-516b1e2f-670f-4591-ba3f-6c412bdd24ff admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:10.692107 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 914/1826] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:10 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:53:10.726762 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-516b1e2f-670f-4591-ba3f-6c412bdd24ff admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:10.733548 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3dcd40-e17d-496e-9d4e-515ac40ac4b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:10.735276 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3dcd40-e17d-496e-9d4e-515ac40ac4b2 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:10.735730 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3dcd40-e17d-496e-9d4e-515ac40ac4b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:10.736162 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3dcd40-e17d-496e-9d4e-515ac40ac4b2 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:10.736966 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a3dcd40-e17d-496e-9d4e-515ac40ac4b2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:10.739076 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a3dcd40-e17d-496e-9d4e-515ac40ac4b2 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:10.752540 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 913/1827] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:53:10 2022] GET /identity/v3/roles => generated 1420 bytes in 129 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 11 00:53:10.756289 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3e8632-d5f9-40e0-8b47-184ce6fd7c93 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:10.771353 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3e8632-d5f9-40e0-8b47-184ce6fd7c93 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:10.778176 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-793b7a7f-cf76-456e-829b-df30f7b13972 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:10.779649 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-793b7a7f-cf76-456e-829b-df30f7b13972 admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:10.780035 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-793b7a7f-cf76-456e-829b-df30f7b13972 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:10.780702 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-793b7a7f-cf76-456e-829b-df30f7b13972 admin admin] PATH_INFO: `/v3/projects/2902b795d287496fbf3688d0feedbc47/users/d2976f8d35684109a2fc6e93cbc547fa/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:10.783384 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-793b7a7f-cf76-456e-829b-df30f7b13972 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2902b795d287496fbf3688d0feedbc47, user_id=d2976f8d35684109a2fc6e93cbc547fa, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:10.787511 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-793b7a7f-cf76-456e-829b-df30f7b13972 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:10.804880 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 915/1828] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:53:10 2022] PUT /identity/v3/projects/2902b795d287496fbf3688d0feedbc47/users/d2976f8d35684109a2fc6e93cbc547fa/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:11.700798 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3dcd40-e17d-496e-9d4e-515ac40ac4b2 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:11.715642 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3dcd40-e17d-496e-9d4e-515ac40ac4b2 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:11.843726 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-793b7a7f-cf76-456e-829b-df30f7b13972 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:11.846472 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-793b7a7f-cf76-456e-829b-df30f7b13972 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:11.870343 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a2c09f0-0d01-4c07-87a8-043c67a4e00a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:11.871938 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2c09f0-0d01-4c07-87a8-043c67a4e00a admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:11.873266 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2c09f0-0d01-4c07-87a8-043c67a4e00a admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:11.874238 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2c09f0-0d01-4c07-87a8-043c67a4e00a admin admin] PATH_INFO: `/v3/users/92417d1a94d540b987e3b9856f753c38` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:11.877705 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a2c09f0-0d01-4c07-87a8-043c67a4e00a admin admin] RBAC: Authorizing `identity:delete_user(user_id=92417d1a94d540b987e3b9856f753c38)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:11.881707 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a2c09f0-0d01-4c07-87a8-043c67a4e00a admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:11.979809 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=93,keystone:INSERT=2,keystone:DELETE=5 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:12.015757 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-2a2c09f0-0d01-4c07-87a8-043c67a4e00a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '92417d1a94d540b987e3b9856f753c38'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:12.016642 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-2a2c09f0-0d01-4c07-87a8-043c67a4e00a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '92417d1a94d540b987e3b9856f753c38'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:12.017418 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-2a2c09f0-0d01-4c07-87a8-043c67a4e00a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '92417d1a94d540b987e3b9856f753c38'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:12.037131 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-2a2c09f0-0d01-4c07-87a8-043c67a4e00a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '92417d1a94d540b987e3b9856f753c38'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:12.061571 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-2a2c09f0-0d01-4c07-87a8-043c67a4e00a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '92417d1a94d540b987e3b9856f753c38'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:12.084349 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 914/1829] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:53:11 2022] DELETE /identity/v3/users/92417d1a94d540b987e3b9856f753c38 => generated 0 bytes in 384 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:12.088470 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9150be7a-002d-40b9-aa20-f4c3966c1c48 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:12.088958 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9150be7a-002d-40b9-aa20-f4c3966c1c48 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:12.089382 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9150be7a-002d-40b9-aa20-f4c3966c1c48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:12.130245 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:12.139585 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-793b7a7f-cf76-456e-829b-df30f7b13972 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:12.149545 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-9150be7a-002d-40b9-aa20-f4c3966c1c48 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:12.150999 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-9150be7a-002d-40b9-aa20-f4c3966c1c48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:12.251016 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 915/1830] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:53:12.324175 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9150be7a-002d-40b9-aa20-f4c3966c1c48 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:12.326268 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-9150be7a-002d-40b9-aa20-f4c3966c1c48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:12.365989 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ec2561-cad6-4ce3-a49e-8cbc369ebb64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:12.366653 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ec2561-cad6-4ce3-a49e-8cbc369ebb64 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:12.367003 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ec2561-cad6-4ce3-a49e-8cbc369ebb64 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:12.367284 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ec2561-cad6-4ce3-a49e-8cbc369ebb64 admin admin] PATH_INFO: `/v3/users/c091fd8fee78476ca15fca9655597677` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:12.373347 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ec2561-cad6-4ce3-a49e-8cbc369ebb64 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c091fd8fee78476ca15fca9655597677)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:12.375530 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ec2561-cad6-4ce3-a49e-8cbc369ebb64 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:12.437867 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9150be7a-002d-40b9-aa20-f4c3966c1c48 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:12.444208 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fbdd420-8e6a-4678-ae4f-90bbd483eb7e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:12.444876 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbdd420-8e6a-4678-ae4f-90bbd483eb7e service neutron] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:12.445181 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbdd420-8e6a-4678-ae4f-90bbd483eb7e service neutron] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:12.445512 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbdd420-8e6a-4678-ae4f-90bbd483eb7e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:12.460797 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fbdd420-8e6a-4678-ae4f-90bbd483eb7e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:12.463303 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fbdd420-8e6a-4678-ae4f-90bbd483eb7e service neutron] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:12.477000 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 916/1831] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:53:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 154 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:53:12.490299 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-97ec2561-cad6-4ce3-a49e-8cbc369ebb64 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c091fd8fee78476ca15fca9655597677'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:12.491462 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-97ec2561-cad6-4ce3-a49e-8cbc369ebb64 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c091fd8fee78476ca15fca9655597677'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:12.492760 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-97ec2561-cad6-4ce3-a49e-8cbc369ebb64 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c091fd8fee78476ca15fca9655597677'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:12.503045 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-97ec2561-cad6-4ce3-a49e-8cbc369ebb64 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c091fd8fee78476ca15fca9655597677'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:12.521854 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-97ec2561-cad6-4ce3-a49e-8cbc369ebb64 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c091fd8fee78476ca15fca9655597677'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:12.554347 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 916/1832] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:53:11 2022] DELETE /identity/v3/users/c091fd8fee78476ca15fca9655597677 => generated 0 bytes in 709 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:12.556775 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f43f5f9-b077-45ce-98ec-11794171c380 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:12.557056 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f43f5f9-b077-45ce-98ec-11794171c380 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:12.557331 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f43f5f9-b077-45ce-98ec-11794171c380 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:12.588535 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-9f43f5f9-b077-45ce-98ec-11794171c380 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:12.594037 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-9f43f5f9-b077-45ce-98ec-11794171c380 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:12.717352 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:12.747340 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 917/1833] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:53:12.823601 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ec2561-cad6-4ce3-a49e-8cbc369ebb64 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:12.825536 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-97ec2561-cad6-4ce3-a49e-8cbc369ebb64 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:13.021882 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ec2561-cad6-4ce3-a49e-8cbc369ebb64 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:13.031254 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ae148e3-d992-47f2-afe4-1f86e1d7211c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:13.036745 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae148e3-d992-47f2-afe4-1f86e1d7211c service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:13.037312 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae148e3-d992-47f2-afe4-1f86e1d7211c service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:13.037772 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae148e3-d992-47f2-afe4-1f86e1d7211c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:13.058106 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ae148e3-d992-47f2-afe4-1f86e1d7211c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:13.059874 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ae148e3-d992-47f2-afe4-1f86e1d7211c service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:13.072086 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 917/1834] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:53:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 249 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:53:13.692547 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e738f8b-4779-4d7c-a69d-bd8936196497 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:13.692880 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e738f8b-4779-4d7c-a69d-bd8936196497 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:13.693109 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e738f8b-4779-4d7c-a69d-bd8936196497 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:13.749994 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-0e738f8b-4779-4d7c-a69d-bd8936196497 None None] MFA Rules not processed for user `d2976f8d35684109a2fc6e93cbc547fa`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:13.752047 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-0e738f8b-4779-4d7c-a69d-bd8936196497 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:13.908457 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ae148e3-d992-47f2-afe4-1f86e1d7211c service neutron] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:13.909982 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-1ae148e3-d992-47f2-afe4-1f86e1d7211c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:13.934704 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 918/1835] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:13 2022] POST /identity/v3/auth/tokens => generated 3347 bytes in 243 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:53:13.953003 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ae148e3-d992-47f2-afe4-1f86e1d7211c service neutron] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:13.964915 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b80d0f1-338a-4e57-80ad-104e780641d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:13.965801 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b80d0f1-338a-4e57-80ad-104e780641d8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:13.966576 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b80d0f1-338a-4e57-80ad-104e780641d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:13.967211 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b80d0f1-338a-4e57-80ad-104e780641d8 admin admin] PATH_INFO: `/v3/projects/94e07bd53dc44496a1bba2e64a36751e` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:13.968580 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b80d0f1-338a-4e57-80ad-104e780641d8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=94e07bd53dc44496a1bba2e64a36751e)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:13.970647 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b80d0f1-338a-4e57-80ad-104e780641d8 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:14.042325 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:14.093635 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-9b80d0f1-338a-4e57-80ad-104e780641d8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '94e07bd53dc44496a1bba2e64a36751e', 'request_id': 'req-9b80d0f1-338a-4e57-80ad-104e780641d8'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:14.103557 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-9b80d0f1-338a-4e57-80ad-104e780641d8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '94e07bd53dc44496a1bba2e64a36751e', 'request_id': 'req-9b80d0f1-338a-4e57-80ad-104e780641d8'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:14.125777 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-9b80d0f1-338a-4e57-80ad-104e780641d8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '94e07bd53dc44496a1bba2e64a36751e', 'request_id': 'req-9b80d0f1-338a-4e57-80ad-104e780641d8'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:14.150009 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-9b80d0f1-338a-4e57-80ad-104e780641d8 admin admin] The token cache is being invalidate because project 94e07bd53dc44496a1bba2e64a36751e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87924) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:53:14.151552 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-9b80d0f1-338a-4e57-80ad-104e780641d8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:14.155482 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-9b80d0f1-338a-4e57-80ad-104e780641d8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:14.163762 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 918/1836] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:53:13 2022] DELETE /identity/v3/projects/94e07bd53dc44496a1bba2e64a36751e => generated 0 bytes in 254 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:14.166942 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e738f8b-4779-4d7c-a69d-bd8936196497 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:14.184033 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e738f8b-4779-4d7c-a69d-bd8936196497 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:14.282620 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bfb92af-c734-4489-84e3-b1bb47745d8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:14.283304 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfb92af-c734-4489-84e3-b1bb47745d8f admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:14.283579 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfb92af-c734-4489-84e3-b1bb47745d8f admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:14.283849 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfb92af-c734-4489-84e3-b1bb47745d8f admin admin] PATH_INFO: `/v3/users/48a1c0c4c3da47e5a13582a2e167e03b` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:14.292458 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b80d0f1-338a-4e57-80ad-104e780641d8 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:14.298147 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-9b80d0f1-338a-4e57-80ad-104e780641d8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:14.303809 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bfb92af-c734-4489-84e3-b1bb47745d8f admin admin] RBAC: Authorizing `identity:delete_user(user_id=48a1c0c4c3da47e5a13582a2e167e03b)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:14.328205 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b80d0f1-338a-4e57-80ad-104e780641d8 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:14.341336 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a90d76-20ff-4144-a410-745402b85d0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:14.343626 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a90d76-20ff-4144-a410-745402b85d0e admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:14.344570 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a90d76-20ff-4144-a410-745402b85d0e admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:14.345048 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a90d76-20ff-4144-a410-745402b85d0e admin admin] PATH_INFO: `/v3/projects/653a37f96cff4020a54b9efc0e393f33` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:14.346646 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bfb92af-c734-4489-84e3-b1bb47745d8f admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:14.347069 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a90d76-20ff-4144-a410-745402b85d0e admin admin] RBAC: Authorizing `identity:delete_project(project_id=653a37f96cff4020a54b9efc0e393f33)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:14.349597 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a90d76-20ff-4144-a410-745402b85d0e admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:14.510762 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-3bfb92af-c734-4489-84e3-b1bb47745d8f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '48a1c0c4c3da47e5a13582a2e167e03b'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:14.511664 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-3bfb92af-c734-4489-84e3-b1bb47745d8f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '48a1c0c4c3da47e5a13582a2e167e03b'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:14.512699 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-3bfb92af-c734-4489-84e3-b1bb47745d8f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '48a1c0c4c3da47e5a13582a2e167e03b'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:14.539543 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-3bfb92af-c734-4489-84e3-b1bb47745d8f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '48a1c0c4c3da47e5a13582a2e167e03b'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:14.564997 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-72a90d76-20ff-4144-a410-745402b85d0e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '653a37f96cff4020a54b9efc0e393f33', 'request_id': 'req-72a90d76-20ff-4144-a410-745402b85d0e'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:14.580797 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-3bfb92af-c734-4489-84e3-b1bb47745d8f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '48a1c0c4c3da47e5a13582a2e167e03b'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:14.595596 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-72a90d76-20ff-4144-a410-745402b85d0e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '653a37f96cff4020a54b9efc0e393f33', 'request_id': 'req-72a90d76-20ff-4144-a410-745402b85d0e'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:14.597586 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:14.606355 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 919/1837] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:53:14 2022] DELETE /identity/v3/users/48a1c0c4c3da47e5a13582a2e167e03b => generated 0 bytes in 440 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:14.611281 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08bae7a1-34c4-40d1-a51a-55a2e61c3664 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:14.612780 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08bae7a1-34c4-40d1-a51a-55a2e61c3664 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:14.613347 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08bae7a1-34c4-40d1-a51a-55a2e61c3664 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:14.614041 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-72a90d76-20ff-4144-a410-745402b85d0e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '653a37f96cff4020a54b9efc0e393f33', 'request_id': 'req-72a90d76-20ff-4144-a410-745402b85d0e'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:14.616446 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 920/1838] 10.222.0.27 () {54 vars in 921 bytes} [Sun Sep 11 00:53:14 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:53:14.634641 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-72a90d76-20ff-4144-a410-745402b85d0e admin admin] The token cache is being invalidate because project 653a37f96cff4020a54b9efc0e393f33 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87924) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:53:14.634995 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-72a90d76-20ff-4144-a410-745402b85d0e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:14.635710 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-72a90d76-20ff-4144-a410-745402b85d0e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:14.640017 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 919/1839] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:53:14 2022] DELETE /identity/v3/projects/653a37f96cff4020a54b9efc0e393f33 => generated 0 bytes in 348 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:14.643526 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08bae7a1-34c4-40d1-a51a-55a2e61c3664 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:14.646031 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-08bae7a1-34c4-40d1-a51a-55a2e61c3664 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:14.674619 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ab61f4-8952-4de0-97bb-7512ea0bc744 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:14.675511 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ab61f4-8952-4de0-97bb-7512ea0bc744 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:14.675511 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ab61f4-8952-4de0-97bb-7512ea0bc744 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:14.709985 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-00ab61f4-8952-4de0-97bb-7512ea0bc744 None None] MFA Rules not processed for user `835def0e6c6d4365a2767051509eb0d6`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:14.711816 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-00ab61f4-8952-4de0-97bb-7512ea0bc744 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:14.774570 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08bae7a1-34c4-40d1-a51a-55a2e61c3664 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:14.782260 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45373697-6974-459f-9e59-4dbb85f7d152 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:14.783358 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45373697-6974-459f-9e59-4dbb85f7d152 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:14.784518 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45373697-6974-459f-9e59-4dbb85f7d152 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:14.784922 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45373697-6974-459f-9e59-4dbb85f7d152 admin admin] PATH_INFO: `/v3/users/08879d8838be40f1b9ed5b59d4e48053` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:14.787586 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45373697-6974-459f-9e59-4dbb85f7d152 admin admin] RBAC: Authorizing `identity:delete_user(user_id=08879d8838be40f1b9ed5b59d4e48053)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:14.790706 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45373697-6974-459f-9e59-4dbb85f7d152 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:14.809877 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 920/1840] 10.222.0.27 () {58 vars in 1018 bytes} [Sun Sep 11 00:53:14 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:53:14.892826 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-45373697-6974-459f-9e59-4dbb85f7d152 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '08879d8838be40f1b9ed5b59d4e48053'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:14.894291 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-45373697-6974-459f-9e59-4dbb85f7d152 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '08879d8838be40f1b9ed5b59d4e48053'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:14.896350 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-45373697-6974-459f-9e59-4dbb85f7d152 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '08879d8838be40f1b9ed5b59d4e48053'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:14.907117 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-45373697-6974-459f-9e59-4dbb85f7d152 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '08879d8838be40f1b9ed5b59d4e48053'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:14.932071 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-45373697-6974-459f-9e59-4dbb85f7d152 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '08879d8838be40f1b9ed5b59d4e48053'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:14.953048 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 921/1841] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:53:14 2022] DELETE /identity/v3/users/08879d8838be40f1b9ed5b59d4e48053 => generated 0 bytes in 309 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:15.205785 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ab61f4-8952-4de0-97bb-7512ea0bc744 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:15.207365 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-00ab61f4-8952-4de0-97bb-7512ea0bc744 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:15.228787 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ab61f4-8952-4de0-97bb-7512ea0bc744 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:15.256681 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:DELETE=14,keystone:SELECT=84,keystone:INSERT=2 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:15.312227 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7154b5b3-27cb-4daa-a19c-5c2e783aa88b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '835def0e6c6d4365a2767051509eb0d6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'ResellerAdmin', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:15.312936 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7154b5b3-27cb-4daa-a19c-5c2e783aa88b service nova] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:15.313285 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7154b5b3-27cb-4daa-a19c-5c2e783aa88b service nova] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:15.313617 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7154b5b3-27cb-4daa-a19c-5c2e783aa88b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:15.316198 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-7154b5b3-27cb-4daa-a19c-5c2e783aa88b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:15.338164 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7154b5b3-27cb-4daa-a19c-5c2e783aa88b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:15.339847 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7154b5b3-27cb-4daa-a19c-5c2e783aa88b service nova] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:15.352951 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 921/1842] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:53:15 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:53:15.790692 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b6370e-6284-456a-a04d-734a7ef4a3ee None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:15.791241 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b6370e-6284-456a-a04d-734a7ef4a3ee None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:15.791418 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b6370e-6284-456a-a04d-734a7ef4a3ee None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:15.792896 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 922/1843] 10.222.0.27 () {54 vars in 921 bytes} [Sun Sep 11 00:53:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:53:15.840980 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f03e8af9-7e1c-41b9-be57-8482d2ab1070 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:15.841220 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f03e8af9-7e1c-41b9-be57-8482d2ab1070 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:15.841544 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f03e8af9-7e1c-41b9-be57-8482d2ab1070 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:15.888068 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-f03e8af9-7e1c-41b9-be57-8482d2ab1070 None None] MFA Rules not processed for user `cdbfc2ccd73e4a5eb8c999be5b094619`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:15.889771 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-f03e8af9-7e1c-41b9-be57-8482d2ab1070 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:16.022269 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 922/1844] 10.222.0.27 () {58 vars in 1018 bytes} [Sun Sep 11 00:53:15 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:53:16.166291 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67b6370e-6284-456a-a04d-734a7ef4a3ee None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:16.168765 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-67b6370e-6284-456a-a04d-734a7ef4a3ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:16.191405 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67b6370e-6284-456a-a04d-734a7ef4a3ee None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:16.201762 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973d8920-5ebf-4ea3-b9ae-9c0ee4277024 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cdbfc2ccd73e4a5eb8c999be5b094619', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:16.203106 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973d8920-5ebf-4ea3-b9ae-9c0ee4277024 service placement] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:16.203593 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973d8920-5ebf-4ea3-b9ae-9c0ee4277024 service placement] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:16.204083 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973d8920-5ebf-4ea3-b9ae-9c0ee4277024 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:16.220020 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-973d8920-5ebf-4ea3-b9ae-9c0ee4277024 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:16.224546 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-973d8920-5ebf-4ea3-b9ae-9c0ee4277024 service placement] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:16.247877 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 923/1845] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:53:16 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:53:16.530896 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f03e8af9-7e1c-41b9-be57-8482d2ab1070 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:16.533854 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-f03e8af9-7e1c-41b9-be57-8482d2ab1070 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:16.550933 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973d8920-5ebf-4ea3-b9ae-9c0ee4277024 service placement] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:16.552521 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-973d8920-5ebf-4ea3-b9ae-9c0ee4277024 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:16.692931 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973d8920-5ebf-4ea3-b9ae-9c0ee4277024 service placement] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:16.698432 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db387d4b-7df3-416b-bd09-8d15cb70c3c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:16.699505 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db387d4b-7df3-416b-bd09-8d15cb70c3c9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:16.700226 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db387d4b-7df3-416b-bd09-8d15cb70c3c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:16.702386 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db387d4b-7df3-416b-bd09-8d15cb70c3c9 admin admin] PATH_INFO: `/v3/projects/fb3a1ee0776c43f4bc17b8b02264fc57` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:16.704062 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db387d4b-7df3-416b-bd09-8d15cb70c3c9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fb3a1ee0776c43f4bc17b8b02264fc57)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:16.706276 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db387d4b-7df3-416b-bd09-8d15cb70c3c9 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:16.716462 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:16.734055 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f03e8af9-7e1c-41b9-be57-8482d2ab1070 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:16.749241 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d142267-25d3-43f7-a59b-97d1dd65eb46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:16.750330 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d142267-25d3-43f7-a59b-97d1dd65eb46 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:16.751177 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d142267-25d3-43f7-a59b-97d1dd65eb46 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:16.752108 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d142267-25d3-43f7-a59b-97d1dd65eb46 admin admin] PATH_INFO: `/v3/projects/cae42aac78b2408b9248ad5176669d3b` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:16.754364 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d142267-25d3-43f7-a59b-97d1dd65eb46 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cae42aac78b2408b9248ad5176669d3b)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:16.757289 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d142267-25d3-43f7-a59b-97d1dd65eb46 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:16.787893 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-db387d4b-7df3-416b-bd09-8d15cb70c3c9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fb3a1ee0776c43f4bc17b8b02264fc57', 'request_id': 'req-db387d4b-7df3-416b-bd09-8d15cb70c3c9'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:16.796690 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-db387d4b-7df3-416b-bd09-8d15cb70c3c9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fb3a1ee0776c43f4bc17b8b02264fc57', 'request_id': 'req-db387d4b-7df3-416b-bd09-8d15cb70c3c9'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:16.804420 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-db387d4b-7df3-416b-bd09-8d15cb70c3c9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fb3a1ee0776c43f4bc17b8b02264fc57', 'request_id': 'req-db387d4b-7df3-416b-bd09-8d15cb70c3c9'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:16.821613 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-db387d4b-7df3-416b-bd09-8d15cb70c3c9 admin admin] The token cache is being invalidate because project fb3a1ee0776c43f4bc17b8b02264fc57 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87923) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:53:16.822251 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-db387d4b-7df3-416b-bd09-8d15cb70c3c9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:16.823286 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-db387d4b-7df3-416b-bd09-8d15cb70c3c9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:16.826472 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 924/1846] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:53:16 2022] DELETE /identity/v3/projects/fb3a1ee0776c43f4bc17b8b02264fc57 => generated 0 bytes in 276 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:16.832370 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db387d4b-7df3-416b-bd09-8d15cb70c3c9 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:16.836293 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:16.839039 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-db387d4b-7df3-416b-bd09-8d15cb70c3c9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:16.844658 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-9d142267-25d3-43f7-a59b-97d1dd65eb46 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cae42aac78b2408b9248ad5176669d3b', 'request_id': 'req-9d142267-25d3-43f7-a59b-97d1dd65eb46'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:16.853922 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-9d142267-25d3-43f7-a59b-97d1dd65eb46 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cae42aac78b2408b9248ad5176669d3b', 'request_id': 'req-9d142267-25d3-43f7-a59b-97d1dd65eb46'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:16.865281 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-9d142267-25d3-43f7-a59b-97d1dd65eb46 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cae42aac78b2408b9248ad5176669d3b', 'request_id': 'req-9d142267-25d3-43f7-a59b-97d1dd65eb46'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:16.883905 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-9d142267-25d3-43f7-a59b-97d1dd65eb46 admin admin] The token cache is being invalidate because project cae42aac78b2408b9248ad5176669d3b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87924) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:53:16.884315 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-9d142267-25d3-43f7-a59b-97d1dd65eb46 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:16.885328 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-9d142267-25d3-43f7-a59b-97d1dd65eb46 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:16.887176 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 923/1847] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:53:16 2022] DELETE /identity/v3/projects/cae42aac78b2408b9248ad5176669d3b => generated 0 bytes in 357 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:16.893485 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1bd6ecb-59a0-48af-a418-aaed8344c8d0 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:16.893990 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1bd6ecb-59a0-48af-a418-aaed8344c8d0 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:16.895175 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1bd6ecb-59a0-48af-a418-aaed8344c8d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:16.919927 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-c1bd6ecb-59a0-48af-a418-aaed8344c8d0 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:16.921318 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-c1bd6ecb-59a0-48af-a418-aaed8344c8d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:16.928348 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db387d4b-7df3-416b-bd09-8d15cb70c3c9 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:17.004344 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 924/1848] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:53:17.006887 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8ba100-4f8d-471c-b974-07f19af70b7c None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:17.007193 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8ba100-4f8d-471c-b974-07f19af70b7c None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:17.007707 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8ba100-4f8d-471c-b974-07f19af70b7c None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:17.009585 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 925/1849] 10.222.0.27 () {54 vars in 921 bytes} [Sun Sep 11 00:53:17 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:53:17.012152 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae58815-b638-4e21-aef4-d788b7166c0c None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:17.012732 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae58815-b638-4e21-aef4-d788b7166c0c None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:17.013657 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae58815-b638-4e21-aef4-d788b7166c0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:17.014681 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57557bb3-8772-49d8-94d4-04dc28246bb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:17.015554 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57557bb3-8772-49d8-94d4-04dc28246bb6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:17.015826 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57557bb3-8772-49d8-94d4-04dc28246bb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:17.016106 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57557bb3-8772-49d8-94d4-04dc28246bb6 admin admin] PATH_INFO: `/v3/users/6a9d7463d5cc4622b17809b3da23c5db` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:17.017550 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57557bb3-8772-49d8-94d4-04dc28246bb6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6a9d7463d5cc4622b17809b3da23c5db)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:17.019526 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57557bb3-8772-49d8-94d4-04dc28246bb6 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:17.048655 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-cae58815-b638-4e21-aef4-d788b7166c0c None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:17.055273 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-cae58815-b638-4e21-aef4-d788b7166c0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:17.070495 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 926/1850] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 57 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:53:17.070495 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e687917d-10a9-4299-934a-3d6a4abb7f51 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:17.070495 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e687917d-10a9-4299-934a-3d6a4abb7f51 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:17.070495 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e687917d-10a9-4299-934a-3d6a4abb7f51 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:17.094749 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:17.097713 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-57557bb3-8772-49d8-94d4-04dc28246bb6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6a9d7463d5cc4622b17809b3da23c5db'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:17.100994 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-57557bb3-8772-49d8-94d4-04dc28246bb6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6a9d7463d5cc4622b17809b3da23c5db'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:17.102095 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-57557bb3-8772-49d8-94d4-04dc28246bb6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6a9d7463d5cc4622b17809b3da23c5db'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:17.110045 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-e687917d-10a9-4299-934a-3d6a4abb7f51 None None] MFA Rules not processed for user `cdbfc2ccd73e4a5eb8c999be5b094619`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:17.111417 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-e687917d-10a9-4299-934a-3d6a4abb7f51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:17.112435 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-57557bb3-8772-49d8-94d4-04dc28246bb6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6a9d7463d5cc4622b17809b3da23c5db'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:17.128067 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-57557bb3-8772-49d8-94d4-04dc28246bb6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6a9d7463d5cc4622b17809b3da23c5db'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:17.164191 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 925/1851] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:53:16 2022] DELETE /identity/v3/users/6a9d7463d5cc4622b17809b3da23c5db => generated 0 bytes in 329 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:17.164191 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57557bb3-8772-49d8-94d4-04dc28246bb6 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:17.167378 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-57557bb3-8772-49d8-94d4-04dc28246bb6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:17.219797 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 927/1852] 10.222.0.27 () {58 vars in 1018 bytes} [Sun Sep 11 00:53:17 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:53:17.220660 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e687917d-10a9-4299-934a-3d6a4abb7f51 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:17.227675 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-e687917d-10a9-4299-934a-3d6a4abb7f51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:17.248933 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e687917d-10a9-4299-934a-3d6a4abb7f51 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:17.315710 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57557bb3-8772-49d8-94d4-04dc28246bb6 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:17.324199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-186bebcc-aac8-4869-8123-80166001adaa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:17.325019 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-186bebcc-aac8-4869-8123-80166001adaa admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:17.326226 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-186bebcc-aac8-4869-8123-80166001adaa admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:17.326914 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-186bebcc-aac8-4869-8123-80166001adaa admin admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:17.328096 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-186bebcc-aac8-4869-8123-80166001adaa admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:53:17.328733 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-186bebcc-aac8-4869-8123-80166001adaa admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:17.331302 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-186bebcc-aac8-4869-8123-80166001adaa admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:17.342113 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-161abd75-7fda-47af-aa54-f83d0fb5f553 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:17.342760 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-161abd75-7fda-47af-aa54-f83d0fb5f553 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:17.343077 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-161abd75-7fda-47af-aa54-f83d0fb5f553 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:17.343340 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-161abd75-7fda-47af-aa54-f83d0fb5f553 admin admin] PATH_INFO: `/v3/users/de25b2dc88d642789944b637157c078c` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:17.344637 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-161abd75-7fda-47af-aa54-f83d0fb5f553 admin admin] RBAC: Authorizing `identity:delete_user(user_id=de25b2dc88d642789944b637157c078c)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:17.347013 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-161abd75-7fda-47af-aa54-f83d0fb5f553 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:17.356459 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=96,keystone:INSERT=1,keystone:DELETE=3 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:17.359447 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 926/1853] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 00:53:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:53:17.360062 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-186bebcc-aac8-4869-8123-80166001adaa admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:17.361748 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-186bebcc-aac8-4869-8123-80166001adaa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:17.427065 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-161abd75-7fda-47af-aa54-f83d0fb5f553 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'de25b2dc88d642789944b637157c078c'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:17.427897 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-161abd75-7fda-47af-aa54-f83d0fb5f553 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'de25b2dc88d642789944b637157c078c'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:17.428592 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-161abd75-7fda-47af-aa54-f83d0fb5f553 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'de25b2dc88d642789944b637157c078c'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:17.439990 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-161abd75-7fda-47af-aa54-f83d0fb5f553 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'de25b2dc88d642789944b637157c078c'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:17.463107 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-161abd75-7fda-47af-aa54-f83d0fb5f553 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'de25b2dc88d642789944b637157c078c'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:17.490133 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 928/1854] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:53:17 2022] DELETE /identity/v3/users/de25b2dc88d642789944b637157c078c => generated 0 bytes in 270 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:17.491161 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-161abd75-7fda-47af-aa54-f83d0fb5f553 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:17.492982 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-161abd75-7fda-47af-aa54-f83d0fb5f553 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:17.494185 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-186bebcc-aac8-4869-8123-80166001adaa admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:17.573347 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=96,keystone:DELETE=3,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:17.583186 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99fc0672-5ae7-45a8-9372-f84a9652a6b7 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cdbfc2ccd73e4a5eb8c999be5b094619', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:17.584107 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99fc0672-5ae7-45a8-9372-f84a9652a6b7 service placement] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:17.584631 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99fc0672-5ae7-45a8-9372-f84a9652a6b7 service placement] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:17.584770 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-161abd75-7fda-47af-aa54-f83d0fb5f553 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:17.585272 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99fc0672-5ae7-45a8-9372-f84a9652a6b7 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:17.587745 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-99fc0672-5ae7-45a8-9372-f84a9652a6b7 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:17.591003 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcd4f572-f512-40be-86ee-dd02da2c1c9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:17.593193 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd4f572-f512-40be-86ee-dd02da2c1c9c admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:17.593531 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd4f572-f512-40be-86ee-dd02da2c1c9c admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:17.593858 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd4f572-f512-40be-86ee-dd02da2c1c9c admin admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:17.594717 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcd4f572-f512-40be-86ee-dd02da2c1c9c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:17.596764 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcd4f572-f512-40be-86ee-dd02da2c1c9c admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:17.604508 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99fc0672-5ae7-45a8-9372-f84a9652a6b7 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:17.606513 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99fc0672-5ae7-45a8-9372-f84a9652a6b7 service placement] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:17.619564 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 927/1855] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:53:17 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 260 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:53:17.641634 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99fc0672-5ae7-45a8-9372-f84a9652a6b7 service placement] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:17.649784 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 929/1856] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:53:17 2022] POST /identity/v3/projects => generated 365 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:53:17.656520 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99fc0672-5ae7-45a8-9372-f84a9652a6b7 service placement] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:17.773935 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcd4f572-f512-40be-86ee-dd02da2c1c9c admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:17.775961 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-dcd4f572-f512-40be-86ee-dd02da2c1c9c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:17.791051 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfee1454-e0ba-487f-a88d-0bd143a7b675 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:17.794038 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfee1454-e0ba-487f-a88d-0bd143a7b675 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:17.795903 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfee1454-e0ba-487f-a88d-0bd143a7b675 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:17.798818 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfee1454-e0ba-487f-a88d-0bd143a7b675 admin admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:17.800720 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfee1454-e0ba-487f-a88d-0bd143a7b675 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:17.807220 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcd4f572-f512-40be-86ee-dd02da2c1c9c admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:17.808120 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfee1454-e0ba-487f-a88d-0bd143a7b675 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:17.821989 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-138fea33-f922-4eff-a905-c588f6541de5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:17.822939 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138fea33-f922-4eff-a905-c588f6541de5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:17.823401 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138fea33-f922-4eff-a905-c588f6541de5 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:17.823991 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138fea33-f922-4eff-a905-c588f6541de5 admin admin] PATH_INFO: `/v3/projects/c124e43066d047a1870389d5e67814d1` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:17.826633 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-138fea33-f922-4eff-a905-c588f6541de5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c124e43066d047a1870389d5e67814d1)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:17.832443 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-138fea33-f922-4eff-a905-c588f6541de5 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:17.853894 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 928/1857] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:53:17 2022] POST /identity/v3/users => generated 283 bytes in 213 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:53:17.858152 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfee1454-e0ba-487f-a88d-0bd143a7b675 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:17.877787 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfee1454-e0ba-487f-a88d-0bd143a7b675 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:17.884575 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e80b6c-2e40-429c-af81-b435bf35022d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:17.885788 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e80b6c-2e40-429c-af81-b435bf35022d admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:17.886485 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e80b6c-2e40-429c-af81-b435bf35022d admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:17.886870 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e80b6c-2e40-429c-af81-b435bf35022d admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:17.887615 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e80b6c-2e40-429c-af81-b435bf35022d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:17.891730 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e80b6c-2e40-429c-af81-b435bf35022d admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:17.914374 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 929/1858] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:53:17 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 11 00:53:17.918443 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e80b6c-2e40-429c-af81-b435bf35022d admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:17.940755 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e80b6c-2e40-429c-af81-b435bf35022d admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:17.953675 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-138fea33-f922-4eff-a905-c588f6541de5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c124e43066d047a1870389d5e67814d1', 'request_id': 'req-138fea33-f922-4eff-a905-c588f6541de5'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:17.963578 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-138fea33-f922-4eff-a905-c588f6541de5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c124e43066d047a1870389d5e67814d1', 'request_id': 'req-138fea33-f922-4eff-a905-c588f6541de5'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:17.974702 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-138fea33-f922-4eff-a905-c588f6541de5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c124e43066d047a1870389d5e67814d1', 'request_id': 'req-138fea33-f922-4eff-a905-c588f6541de5'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:17.984995 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=81,keystone:DELETE=16,keystone:INSERT=3 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:18.005673 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-138fea33-f922-4eff-a905-c588f6541de5 admin admin] The token cache is being invalidate because project c124e43066d047a1870389d5e67814d1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87924) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:53:18.006444 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-138fea33-f922-4eff-a905-c588f6541de5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:18.008182 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-138fea33-f922-4eff-a905-c588f6541de5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:18.010635 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 930/1859] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:53:17 2022] DELETE /identity/v3/projects/c124e43066d047a1870389d5e67814d1 => generated 0 bytes in 237 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:53:18.017331 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2baa2b-99b2-4307-9914-a5a790e2c62e None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:18.017622 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2baa2b-99b2-4307-9914-a5a790e2c62e None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:18.020281 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2baa2b-99b2-4307-9914-a5a790e2c62e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:18.051789 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-fe2baa2b-99b2-4307-9914-a5a790e2c62e None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:18.064009 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-fe2baa2b-99b2-4307-9914-a5a790e2c62e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:18.083291 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58feabec-824f-43ca-9014-aabbd379ae94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:18.083930 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58feabec-824f-43ca-9014-aabbd379ae94 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:18.084213 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58feabec-824f-43ca-9014-aabbd379ae94 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:18.084465 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 931/1860] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 68 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:53:18.084576 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58feabec-824f-43ca-9014-aabbd379ae94 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:18.085135 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58feabec-824f-43ca-9014-aabbd379ae94 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:18.085658 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe2baa2b-99b2-4307-9914-a5a790e2c62e None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:18.087031 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58feabec-824f-43ca-9014-aabbd379ae94 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:18.087408 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-fe2baa2b-99b2-4307-9914-a5a790e2c62e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:18.109605 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe2baa2b-99b2-4307-9914-a5a790e2c62e None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:18.111797 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 930/1861] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:53:17 2022] GET /identity/v3/roles => generated 1420 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:53:18.113235 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16be1a27-744d-4fb1-8012-4ff6b9302e60 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:18.113528 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16be1a27-744d-4fb1-8012-4ff6b9302e60 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:18.114075 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16be1a27-744d-4fb1-8012-4ff6b9302e60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:18.127619 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4bbd4bb-dae9-4210-a156-5b1d59425a31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:18.128558 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4bbd4bb-dae9-4210-a156-5b1d59425a31 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:18.128746 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4bbd4bb-dae9-4210-a156-5b1d59425a31 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:18.129079 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4bbd4bb-dae9-4210-a156-5b1d59425a31 admin admin] PATH_INFO: `/v3/projects/9bcc6857806944d9a22e0f4d53674801` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:18.131946 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4bbd4bb-dae9-4210-a156-5b1d59425a31 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9bcc6857806944d9a22e0f4d53674801)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:18.136578 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4bbd4bb-dae9-4210-a156-5b1d59425a31 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:18.143030 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-16be1a27-744d-4fb1-8012-4ff6b9302e60 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:18.144619 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-16be1a27-744d-4fb1-8012-4ff6b9302e60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:18.154123 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 931/1862] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:53:18.155442 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16be1a27-744d-4fb1-8012-4ff6b9302e60 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:18.156934 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-16be1a27-744d-4fb1-8012-4ff6b9302e60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:18.169970 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16be1a27-744d-4fb1-8012-4ff6b9302e60 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:18.175452 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f4e1a5-2a27-43dc-84cb-d53e84f163c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:18.176426 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f4e1a5-2a27-43dc-84cb-d53e84f163c6 admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:18.176861 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f4e1a5-2a27-43dc-84cb-d53e84f163c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:18.177282 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f4e1a5-2a27-43dc-84cb-d53e84f163c6 admin admin] PATH_INFO: `/v3/projects/afbed9f160d8407598fa6257b82c9c28/users/de3d4478bbb1418dbbb8c048a650d8c4/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:18.196394 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8f4e1a5-2a27-43dc-84cb-d53e84f163c6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=afbed9f160d8407598fa6257b82c9c28, user_id=de3d4478bbb1418dbbb8c048a650d8c4, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:18.199930 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8f4e1a5-2a27-43dc-84cb-d53e84f163c6 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:18.231260 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 932/1863] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:53:18 2022] PUT /identity/v3/projects/afbed9f160d8407598fa6257b82c9c28/users/de3d4478bbb1418dbbb8c048a650d8c4/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 76 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:18.237385 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f4e1a5-2a27-43dc-84cb-d53e84f163c6 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:18.255930 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-f4bbd4bb-dae9-4210-a156-5b1d59425a31 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9bcc6857806944d9a22e0f4d53674801', 'request_id': 'req-f4bbd4bb-dae9-4210-a156-5b1d59425a31'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:18.265864 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-f4bbd4bb-dae9-4210-a156-5b1d59425a31 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9bcc6857806944d9a22e0f4d53674801', 'request_id': 'req-f4bbd4bb-dae9-4210-a156-5b1d59425a31'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:18.276563 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-f4bbd4bb-dae9-4210-a156-5b1d59425a31 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9bcc6857806944d9a22e0f4d53674801', 'request_id': 'req-f4bbd4bb-dae9-4210-a156-5b1d59425a31'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:18.283955 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:18.300676 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-f4bbd4bb-dae9-4210-a156-5b1d59425a31 admin admin] The token cache is being invalidate because project 9bcc6857806944d9a22e0f4d53674801 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87924) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:53:18.301040 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-f4bbd4bb-dae9-4210-a156-5b1d59425a31 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:18.302859 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-f4bbd4bb-dae9-4210-a156-5b1d59425a31 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:18.309070 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 932/1864] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:53:18 2022] DELETE /identity/v3/projects/9bcc6857806944d9a22e0f4d53674801 => generated 0 bytes in 221 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:18.309070 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d331192-e77a-4c54-bd7f-22881dff3b7b None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:18.309070 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d331192-e77a-4c54-bd7f-22881dff3b7b None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:18.309070 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d331192-e77a-4c54-bd7f-22881dff3b7b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:18.367574 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f4e1a5-2a27-43dc-84cb-d53e84f163c6 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:18.371631 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-3d331192-e77a-4c54-bd7f-22881dff3b7b None None] MFA Rules not processed for user `de3d4478bbb1418dbbb8c048a650d8c4`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:18.374283 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-3d331192-e77a-4c54-bd7f-22881dff3b7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:18.475718 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 933/1865] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:18 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:53:18.476411 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d331192-e77a-4c54-bd7f-22881dff3b7b None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:18.477685 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-3d331192-e77a-4c54-bd7f-22881dff3b7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:18.487984 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a6bf956-0e00-4edb-8ded-6ec73d963756 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:18.489521 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6bf956-0e00-4edb-8ded-6ec73d963756 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:18.490106 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6bf956-0e00-4edb-8ded-6ec73d963756 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:18.491136 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6bf956-0e00-4edb-8ded-6ec73d963756 admin admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:18.491928 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a6bf956-0e00-4edb-8ded-6ec73d963756 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:53:18.492361 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a6bf956-0e00-4edb-8ded-6ec73d963756 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:18.494659 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a6bf956-0e00-4edb-8ded-6ec73d963756 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:18.523312 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 933/1866] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 00:53:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 292 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:53:18.527023 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a6bf956-0e00-4edb-8ded-6ec73d963756 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:18.531258 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-8a6bf956-0e00-4edb-8ded-6ec73d963756 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:18.550422 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a6bf956-0e00-4edb-8ded-6ec73d963756 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:18.557864 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3870f4da-55c3-4d36-816d-54be9b7f1312 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:18.558490 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3870f4da-55c3-4d36-816d-54be9b7f1312 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:18.558773 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3870f4da-55c3-4d36-816d-54be9b7f1312 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:18.559082 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3870f4da-55c3-4d36-816d-54be9b7f1312 admin admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:18.559728 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3870f4da-55c3-4d36-816d-54be9b7f1312 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:18.561770 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3870f4da-55c3-4d36-816d-54be9b7f1312 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:18.580068 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d331192-e77a-4c54-bd7f-22881dff3b7b None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:18.587674 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6daf14b2-c46f-423f-979e-6c973304c8eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:18.588399 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6daf14b2-c46f-423f-979e-6c973304c8eb admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:18.589013 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6daf14b2-c46f-423f-979e-6c973304c8eb admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:18.589330 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6daf14b2-c46f-423f-979e-6c973304c8eb admin admin] PATH_INFO: `/v3/users/d2976f8d35684109a2fc6e93cbc547fa` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:18.592016 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6daf14b2-c46f-423f-979e-6c973304c8eb admin admin] RBAC: Authorizing `identity:delete_user(user_id=d2976f8d35684109a2fc6e93cbc547fa)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:18.594564 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6daf14b2-c46f-423f-979e-6c973304c8eb admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:18.596734 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:DELETE=8,keystone:SELECT=91,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:18.624184 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 934/1867] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:53:18 2022] POST /identity/v3/projects => generated 363 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:53:18.626608 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3870f4da-55c3-4d36-816d-54be9b7f1312 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:18.633378 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-3870f4da-55c3-4d36-816d-54be9b7f1312 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:18.710503 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-6daf14b2-c46f-423f-979e-6c973304c8eb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd2976f8d35684109a2fc6e93cbc547fa'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:18.711243 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-6daf14b2-c46f-423f-979e-6c973304c8eb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd2976f8d35684109a2fc6e93cbc547fa'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:18.714993 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-6daf14b2-c46f-423f-979e-6c973304c8eb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd2976f8d35684109a2fc6e93cbc547fa'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:18.726393 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-6daf14b2-c46f-423f-979e-6c973304c8eb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd2976f8d35684109a2fc6e93cbc547fa'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:18.746762 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3870f4da-55c3-4d36-816d-54be9b7f1312 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:18.748393 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-6daf14b2-c46f-423f-979e-6c973304c8eb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd2976f8d35684109a2fc6e93cbc547fa'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:18.752316 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-442d0952-9556-44b5-9085-6f56a86aae28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:18.752855 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442d0952-9556-44b5-9085-6f56a86aae28 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:18.753136 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442d0952-9556-44b5-9085-6f56a86aae28 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:18.753403 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442d0952-9556-44b5-9085-6f56a86aae28 admin admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:18.754049 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-442d0952-9556-44b5-9085-6f56a86aae28 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:18.756008 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-442d0952-9556-44b5-9085-6f56a86aae28 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:18.767575 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 934/1868] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:53:18 2022] DELETE /identity/v3/users/d2976f8d35684109a2fc6e93cbc547fa => generated 0 bytes in 292 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:18.772806 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6daf14b2-c46f-423f-979e-6c973304c8eb admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:18.774169 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-6daf14b2-c46f-423f-979e-6c973304c8eb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:18.799986 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 935/1869] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:53:18 2022] POST /identity/v3/projects => generated 365 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:53:18.804599 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-442d0952-9556-44b5-9085-6f56a86aae28 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:18.810365 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-442d0952-9556-44b5-9085-6f56a86aae28 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:18.874056 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6daf14b2-c46f-423f-979e-6c973304c8eb admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:18.907696 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:18.938218 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-442d0952-9556-44b5-9085-6f56a86aae28 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:18.944522 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3526aed3-5a8e-4008-b773-b69c876165b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:18.945270 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3526aed3-5a8e-4008-b773-b69c876165b9 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:18.945676 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3526aed3-5a8e-4008-b773-b69c876165b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:18.946203 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3526aed3-5a8e-4008-b773-b69c876165b9 admin admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:18.947020 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3526aed3-5a8e-4008-b773-b69c876165b9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:18.951409 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3526aed3-5a8e-4008-b773-b69c876165b9 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:18.990061 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 936/1870] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:53:18 2022] POST /identity/v3/users => generated 283 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:53:18.995868 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3526aed3-5a8e-4008-b773-b69c876165b9 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:19.018508 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fbc59ba-7768-411d-9ca4-e0c2344b7231 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:19.019219 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbc59ba-7768-411d-9ca4-e0c2344b7231 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:19.021569 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbc59ba-7768-411d-9ca4-e0c2344b7231 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:19.023072 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbc59ba-7768-411d-9ca4-e0c2344b7231 admin admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:19.024527 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fbc59ba-7768-411d-9ca4-e0c2344b7231 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:19.026943 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fbc59ba-7768-411d-9ca4-e0c2344b7231 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:19.029112 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3526aed3-5a8e-4008-b773-b69c876165b9 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:19.045428 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a559c506-3078-4d8e-b0c1-91ed63038ddf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:19.046469 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a559c506-3078-4d8e-b0c1-91ed63038ddf admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:19.048244 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a559c506-3078-4d8e-b0c1-91ed63038ddf admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:19.048697 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a559c506-3078-4d8e-b0c1-91ed63038ddf admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:19.049501 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a559c506-3078-4d8e-b0c1-91ed63038ddf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:19.053697 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a559c506-3078-4d8e-b0c1-91ed63038ddf admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:19.058520 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 935/1871] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:53:18 2022] POST /identity/v3/users => generated 288 bytes in 287 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:53:19.065518 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fbc59ba-7768-411d-9ca4-e0c2344b7231 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:19.075020 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 937/1872] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:53:18 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:53:19.080512 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a559c506-3078-4d8e-b0c1-91ed63038ddf admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:19.095477 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fbc59ba-7768-411d-9ca4-e0c2344b7231 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:19.097400 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a559c506-3078-4d8e-b0c1-91ed63038ddf admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:19.103685 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387c0536-5df7-4e85-88da-83d3fdb1a292 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:19.104027 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387c0536-5df7-4e85-88da-83d3fdb1a292 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:19.104308 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387c0536-5df7-4e85-88da-83d3fdb1a292 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:19.104576 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387c0536-5df7-4e85-88da-83d3fdb1a292 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:19.107637 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-387c0536-5df7-4e85-88da-83d3fdb1a292 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:19.109008 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-387c0536-5df7-4e85-88da-83d3fdb1a292 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:19.113252 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc720ddc-be87-4a32-8606-93331dcb7549 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:19.115287 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc720ddc-be87-4a32-8606-93331dcb7549 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:19.116635 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc720ddc-be87-4a32-8606-93331dcb7549 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:19.117077 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc720ddc-be87-4a32-8606-93331dcb7549 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:19.117910 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc720ddc-be87-4a32-8606-93331dcb7549 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:19.119722 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc720ddc-be87-4a32-8606-93331dcb7549 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:19.136784 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 938/1873] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:53:19 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:53:19.138602 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387c0536-5df7-4e85-88da-83d3fdb1a292 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:19.158839 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387c0536-5df7-4e85-88da-83d3fdb1a292 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:19.161194 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 936/1874] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:53:19 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:53:19.164607 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc720ddc-be87-4a32-8606-93331dcb7549 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:19.169827 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de17916a-aa48-4959-bd06-67a9b74bf68b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:19.170527 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de17916a-aa48-4959-bd06-67a9b74bf68b admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:19.170782 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de17916a-aa48-4959-bd06-67a9b74bf68b admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:19.171085 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de17916a-aa48-4959-bd06-67a9b74bf68b admin admin] PATH_INFO: `/v3/projects/f1929149e6454ceab2bbb044f38b711a/users/6e566054861c4da7af7b4f8d3c5464c2/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:19.172813 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-bc720ddc-be87-4a32-8606-93331dcb7549 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:19.196908 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de17916a-aa48-4959-bd06-67a9b74bf68b admin admin] RBAC: Authorizing `identity:create_grant(project_id=f1929149e6454ceab2bbb044f38b711a, user_id=6e566054861c4da7af7b4f8d3c5464c2, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:19.197609 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:19.202634 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de17916a-aa48-4959-bd06-67a9b74bf68b admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:19.206799 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc720ddc-be87-4a32-8606-93331dcb7549 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:19.217685 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a8d0a5-8ce0-44d6-9871-80753bdffcf6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:19.218581 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a8d0a5-8ce0-44d6-9871-80753bdffcf6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:19.218706 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a8d0a5-8ce0-44d6-9871-80753bdffcf6 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:19.220231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a8d0a5-8ce0-44d6-9871-80753bdffcf6 admin admin] PATH_INFO: `/v3/projects/2902b795d287496fbf3688d0feedbc47` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:19.221820 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5a8d0a5-8ce0-44d6-9871-80753bdffcf6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2902b795d287496fbf3688d0feedbc47)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:19.223944 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5a8d0a5-8ce0-44d6-9871-80753bdffcf6 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:19.242874 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 939/1875] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:53:19 2022] PUT /identity/v3/projects/f1929149e6454ceab2bbb044f38b711a/users/6e566054861c4da7af7b4f8d3c5464c2/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 92 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:19.244067 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de17916a-aa48-4959-bd06-67a9b74bf68b admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:19.290957 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-b5a8d0a5-8ce0-44d6-9871-80753bdffcf6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2902b795d287496fbf3688d0feedbc47', 'request_id': 'req-b5a8d0a5-8ce0-44d6-9871-80753bdffcf6'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:19.300197 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-b5a8d0a5-8ce0-44d6-9871-80753bdffcf6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2902b795d287496fbf3688d0feedbc47', 'request_id': 'req-b5a8d0a5-8ce0-44d6-9871-80753bdffcf6'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:19.310048 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-b5a8d0a5-8ce0-44d6-9871-80753bdffcf6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2902b795d287496fbf3688d0feedbc47', 'request_id': 'req-b5a8d0a5-8ce0-44d6-9871-80753bdffcf6'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:19.323962 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-b5a8d0a5-8ce0-44d6-9871-80753bdffcf6 admin admin] The token cache is being invalidate because project 2902b795d287496fbf3688d0feedbc47 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87924) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:53:19.324281 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-b5a8d0a5-8ce0-44d6-9871-80753bdffcf6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:19.327247 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-b5a8d0a5-8ce0-44d6-9871-80753bdffcf6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:19.329576 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de17916a-aa48-4959-bd06-67a9b74bf68b admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:19.330892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 937/1876] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:53:19 2022] DELETE /identity/v3/projects/2902b795d287496fbf3688d0feedbc47 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:19.331795 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc94d00-0270-41eb-80db-dc9d84ab8aa7 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:19.332081 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc94d00-0270-41eb-80db-dc9d84ab8aa7 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:19.332408 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc94d00-0270-41eb-80db-dc9d84ab8aa7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:19.376879 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-adc94d00-0270-41eb-80db-dc9d84ab8aa7 None None] MFA Rules not processed for user `6e566054861c4da7af7b4f8d3c5464c2`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:19.378290 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-adc94d00-0270-41eb-80db-dc9d84ab8aa7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:19.405250 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc94d97-f56a-4bc3-b9bb-54f6b516883d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:19.405925 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc94d97-f56a-4bc3-b9bb-54f6b516883d admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:19.406441 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc94d97-f56a-4bc3-b9bb-54f6b516883d admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:19.406886 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc94d97-f56a-4bc3-b9bb-54f6b516883d admin admin] PATH_INFO: `/v3/projects/4c08a4f7d5f844e8af87c9c0d22cdfa3/users/b0409468f7a140ca9b7ab2cba292c84a/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:19.428901 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc94d97-f56a-4bc3-b9bb-54f6b516883d admin admin] RBAC: Authorizing `identity:create_grant(project_id=4c08a4f7d5f844e8af87c9c0d22cdfa3, user_id=b0409468f7a140ca9b7ab2cba292c84a, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:19.432109 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc94d97-f56a-4bc3-b9bb-54f6b516883d admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:19.448492 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 940/1877] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:53:19 2022] PUT /identity/v3/projects/4c08a4f7d5f844e8af87c9c0d22cdfa3/users/b0409468f7a140ca9b7ab2cba292c84a/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:19.450217 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2141b205-6370-4731-b897-f31fe7650643 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:19.451031 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2141b205-6370-4731-b897-f31fe7650643 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:19.451640 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2141b205-6370-4731-b897-f31fe7650643 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:19.468059 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 938/1878] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:19 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:53:19.474229 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adc94d00-0270-41eb-80db-dc9d84ab8aa7 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:19.486335 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-adc94d00-0270-41eb-80db-dc9d84ab8aa7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:19.491904 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-2141b205-6370-4731-b897-f31fe7650643 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:19.496067 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-2141b205-6370-4731-b897-f31fe7650643 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:19.504810 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:19.511974 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adc94d00-0270-41eb-80db-dc9d84ab8aa7 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:19.577722 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 941/1879] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:53:19.578571 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2141b205-6370-4731-b897-f31fe7650643 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:19.579914 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-2141b205-6370-4731-b897-f31fe7650643 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:19.592358 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce40b37f-8989-439d-a21b-d13fb3a80596 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:19.593104 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce40b37f-8989-439d-a21b-d13fb3a80596 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:19.593352 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce40b37f-8989-439d-a21b-d13fb3a80596 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:19.593536 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce40b37f-8989-439d-a21b-d13fb3a80596 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:19.594535 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce40b37f-8989-439d-a21b-d13fb3a80596 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:19.597150 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce40b37f-8989-439d-a21b-d13fb3a80596 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:19.613726 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2141b205-6370-4731-b897-f31fe7650643 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:19.636610 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 939/1880] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:53:19 2022] GET /identity/v3/roles => generated 1420 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:53:19.636610 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca406834-79af-4a30-9e7d-e4a089793f36 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:19.636610 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca406834-79af-4a30-9e7d-e4a089793f36 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:19.636610 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca406834-79af-4a30-9e7d-e4a089793f36 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:19.643961 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e5f6d18-120a-43ca-9026-4b00607d2714 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:19.644593 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5f6d18-120a-43ca-9026-4b00607d2714 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:19.644872 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5f6d18-120a-43ca-9026-4b00607d2714 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:19.645145 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5f6d18-120a-43ca-9026-4b00607d2714 admin admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:19.645838 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e5f6d18-120a-43ca-9026-4b00607d2714 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:19.648386 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e5f6d18-120a-43ca-9026-4b00607d2714 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:19.674713 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-ca406834-79af-4a30-9e7d-e4a089793f36 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:19.675902 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-ca406834-79af-4a30-9e7d-e4a089793f36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:19.703131 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 940/1881] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 69 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:53:19.704828 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca406834-79af-4a30-9e7d-e4a089793f36 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:19.712538 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 942/1882] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:53:19 2022] POST /identity/v3/projects => generated 367 bytes in 134 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:53:19.713704 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e5f6d18-120a-43ca-9026-4b00607d2714 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:19.720584 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca406834-79af-4a30-9e7d-e4a089793f36 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:19.724473 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:19.856003 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e9a756-a751-409d-bb65-14b6d95d6486 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:19.856003 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e9a756-a751-409d-bb65-14b6d95d6486 admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:19.856247 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e9a756-a751-409d-bb65-14b6d95d6486 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:19.858165 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e9a756-a751-409d-bb65-14b6d95d6486 admin admin] PATH_INFO: `/v3/projects/4c08a4f7d5f844e8af87c9c0d22cdfa3/users/b0409468f7a140ca9b7ab2cba292c84a/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:19.862778 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e5f6d18-120a-43ca-9026-4b00607d2714 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:19.864149 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19e9a756-a751-409d-bb65-14b6d95d6486 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4c08a4f7d5f844e8af87c9c0d22cdfa3, user_id=b0409468f7a140ca9b7ab2cba292c84a, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:19.867695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19e9a756-a751-409d-bb65-14b6d95d6486 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:19.878063 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-253c3448-9eee-4daf-ad93-27ab0a1aa0fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:19.878818 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-253c3448-9eee-4daf-ad93-27ab0a1aa0fe admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:19.879182 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-253c3448-9eee-4daf-ad93-27ab0a1aa0fe admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:19.879692 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-253c3448-9eee-4daf-ad93-27ab0a1aa0fe admin admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:19.880421 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-253c3448-9eee-4daf-ad93-27ab0a1aa0fe admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:53:19.881523 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-253c3448-9eee-4daf-ad93-27ab0a1aa0fe admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:19.883666 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-253c3448-9eee-4daf-ad93-27ab0a1aa0fe admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:19.893834 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 941/1883] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:53:19 2022] PUT /identity/v3/projects/4c08a4f7d5f844e8af87c9c0d22cdfa3/users/b0409468f7a140ca9b7ab2cba292c84a/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:19.898687 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e9a756-a751-409d-bb65-14b6d95d6486 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:19.927676 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 943/1884] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 00:53:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:53:19.932131 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ae09e3-51ff-4f74-b615-cc7e6331183a None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:19.933835 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ae09e3-51ff-4f74-b615-cc7e6331183a None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:19.934124 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ae09e3-51ff-4f74-b615-cc7e6331183a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:20.009540 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-57ae09e3-51ff-4f74-b615-cc7e6331183a None None] MFA Rules not processed for user `b0409468f7a140ca9b7ab2cba292c84a`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:20.011019 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-57ae09e3-51ff-4f74-b615-cc7e6331183a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:20.063895 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e9a756-a751-409d-bb65-14b6d95d6486 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:20.070922 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37163305-cf2d-4ac4-b106-ca7fe960e103 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:20.071922 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37163305-cf2d-4ac4-b106-ca7fe960e103 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:20.072187 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37163305-cf2d-4ac4-b106-ca7fe960e103 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:20.072497 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37163305-cf2d-4ac4-b106-ca7fe960e103 admin admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:20.073472 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37163305-cf2d-4ac4-b106-ca7fe960e103 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:20.076648 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37163305-cf2d-4ac4-b106-ca7fe960e103 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:20.138971 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 942/1885] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:53:19 2022] POST /identity/v3/users => generated 290 bytes in 245 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:53:20.141409 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37163305-cf2d-4ac4-b106-ca7fe960e103 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:20.201326 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37163305-cf2d-4ac4-b106-ca7fe960e103 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:20.215479 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bafa659c-b1c2-45ed-b6dc-fff59b63c18e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:20.216727 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bafa659c-b1c2-45ed-b6dc-fff59b63c18e admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:20.217009 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bafa659c-b1c2-45ed-b6dc-fff59b63c18e admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:20.217281 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bafa659c-b1c2-45ed-b6dc-fff59b63c18e admin admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:20.225326 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 944/1886] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:19 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 293 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:53:20.225493 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bafa659c-b1c2-45ed-b6dc-fff59b63c18e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:20.225613 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57ae09e3-51ff-4f74-b615-cc7e6331183a None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:20.225694 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bafa659c-b1c2-45ed-b6dc-fff59b63c18e admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:20.239936 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57ae09e3-51ff-4f74-b615-cc7e6331183a None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:20.244976 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b443555e-19c4-46ab-9688-0fb0f9bfe9d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:20.245791 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b443555e-19c4-46ab-9688-0fb0f9bfe9d2 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:20.246195 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b443555e-19c4-46ab-9688-0fb0f9bfe9d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:20.246584 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b443555e-19c4-46ab-9688-0fb0f9bfe9d2 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:20.247660 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b443555e-19c4-46ab-9688-0fb0f9bfe9d2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:20.249688 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b443555e-19c4-46ab-9688-0fb0f9bfe9d2 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:20.252365 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:20.267483 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 945/1887] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:53:20 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:53:20.269042 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b443555e-19c4-46ab-9688-0fb0f9bfe9d2 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:20.270962 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-b443555e-19c4-46ab-9688-0fb0f9bfe9d2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:20.319147 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 943/1888] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:53:20 2022] POST /identity/v3/projects => generated 375 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:53:20.324253 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bafa659c-b1c2-45ed-b6dc-fff59b63c18e admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:20.398101 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b443555e-19c4-46ab-9688-0fb0f9bfe9d2 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:20.451867 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bafa659c-b1c2-45ed-b6dc-fff59b63c18e admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:20.458698 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f1dc8f6-267b-4169-89bc-1bf706ba1c30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:20.459932 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1dc8f6-267b-4169-89bc-1bf706ba1c30 admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:20.460298 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1dc8f6-267b-4169-89bc-1bf706ba1c30 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:20.460673 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1dc8f6-267b-4169-89bc-1bf706ba1c30 admin admin] PATH_INFO: `/v3/projects/f7ad2cb683ef49d1bd4697a38369b280/users/3b341df142a04390938460ec5bd0718f/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:20.479386 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:20.486255 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f1dc8f6-267b-4169-89bc-1bf706ba1c30 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f7ad2cb683ef49d1bd4697a38369b280, user_id=3b341df142a04390938460ec5bd0718f, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:20.489324 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f1dc8f6-267b-4169-89bc-1bf706ba1c30 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:20.514009 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 944/1889] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:53:20 2022] PUT /identity/v3/projects/f7ad2cb683ef49d1bd4697a38369b280/users/3b341df142a04390938460ec5bd0718f/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:20.515957 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f1dc8f6-267b-4169-89bc-1bf706ba1c30 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:20.554942 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d2279e4-34c2-4a83-b170-b593b68dd8bb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:20.555570 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d2279e4-34c2-4a83-b170-b593b68dd8bb service cinder] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:20.555850 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d2279e4-34c2-4a83-b170-b593b68dd8bb service cinder] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:20.556119 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d2279e4-34c2-4a83-b170-b593b68dd8bb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:20.616079 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f1dc8f6-267b-4169-89bc-1bf706ba1c30 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:20.622139 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6246afb8-6c3b-4ce7-ac3c-8ddcdf4e4665 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:20.624371 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6246afb8-6c3b-4ce7-ac3c-8ddcdf4e4665 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:20.625314 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6246afb8-6c3b-4ce7-ac3c-8ddcdf4e4665 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:20.625694 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6246afb8-6c3b-4ce7-ac3c-8ddcdf4e4665 admin admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:20.627230 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6246afb8-6c3b-4ce7-ac3c-8ddcdf4e4665 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:20.629216 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6246afb8-6c3b-4ce7-ac3c-8ddcdf4e4665 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:20.670936 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 945/1890] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:53:20 2022] POST /identity/v3/users => generated 288 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:53:20.675001 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6246afb8-6c3b-4ce7-ac3c-8ddcdf4e4665 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:20.700089 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d2279e4-34c2-4a83-b170-b593b68dd8bb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:20.701683 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d2279e4-34c2-4a83-b170-b593b68dd8bb service cinder] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:20.701810 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6246afb8-6c3b-4ce7-ac3c-8ddcdf4e4665 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:20.713988 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bb9a58b-ebff-434b-a1a7-c045e93f404d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:20.714891 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb9a58b-ebff-434b-a1a7-c045e93f404d admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:20.715665 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb9a58b-ebff-434b-a1a7-c045e93f404d admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:20.715665 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb9a58b-ebff-434b-a1a7-c045e93f404d admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:20.716388 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bb9a58b-ebff-434b-a1a7-c045e93f404d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:20.720945 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bb9a58b-ebff-434b-a1a7-c045e93f404d admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:20.724700 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 946/1891] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:53:20 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 457 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:53:20.725560 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d2279e4-34c2-4a83-b170-b593b68dd8bb service cinder] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:20.740557 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d2279e4-34c2-4a83-b170-b593b68dd8bb service cinder] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:20.746201 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49922fa-c62d-4622-8dd3-46ac03707316 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:20.746943 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49922fa-c62d-4622-8dd3-46ac03707316 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:20.748233 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49922fa-c62d-4622-8dd3-46ac03707316 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:20.748523 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49922fa-c62d-4622-8dd3-46ac03707316 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:20.749220 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 946/1892] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:53:20 2022] GET /identity/v3/roles => generated 1420 bytes in 77 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:53:20.759078 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bb9a58b-ebff-434b-a1a7-c045e93f404d admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:20.759402 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e49922fa-c62d-4622-8dd3-46ac03707316 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:20.759402 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e49922fa-c62d-4622-8dd3-46ac03707316 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:20.776092 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bb9a58b-ebff-434b-a1a7-c045e93f404d admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:20.792551 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85f8653a-5baf-437f-bda8-2fe5620ae43f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:20.793495 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f8653a-5baf-437f-bda8-2fe5620ae43f admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:20.793937 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f8653a-5baf-437f-bda8-2fe5620ae43f admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:20.794837 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f8653a-5baf-437f-bda8-2fe5620ae43f admin admin] PATH_INFO: `/v3/projects/f7ad2cb683ef49d1bd4697a38369b280/users/3b341df142a04390938460ec5bd0718f/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:20.800566 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85f8653a-5baf-437f-bda8-2fe5620ae43f admin admin] RBAC: Authorizing `identity:create_grant(project_id=f7ad2cb683ef49d1bd4697a38369b280, user_id=3b341df142a04390938460ec5bd0718f, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:20.803645 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85f8653a-5baf-437f-bda8-2fe5620ae43f admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:20.833083 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 947/1893] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:53:20 2022] GET /identity/v3/roles => generated 1420 bytes in 108 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:53:20.834557 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49922fa-c62d-4622-8dd3-46ac03707316 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:20.836111 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 947/1894] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:53:20 2022] PUT /identity/v3/projects/f7ad2cb683ef49d1bd4697a38369b280/users/3b341df142a04390938460ec5bd0718f/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 83 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:20.846713 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d464ba-33f9-426e-8099-80b5b2ff81a5 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:20.847179 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d464ba-33f9-426e-8099-80b5b2ff81a5 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:20.847639 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d464ba-33f9-426e-8099-80b5b2ff81a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:20.883241 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49922fa-c62d-4622-8dd3-46ac03707316 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:20.908108 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-29d464ba-33f9-426e-8099-80b5b2ff81a5 None None] MFA Rules not processed for user `3b341df142a04390938460ec5bd0718f`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:20.909340 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-29d464ba-33f9-426e-8099-80b5b2ff81a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:20.971824 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:20.985458 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00eb1b4d-a061-4277-a100-bf3f85cadd92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:20.986930 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00eb1b4d-a061-4277-a100-bf3f85cadd92 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:20.987390 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00eb1b4d-a061-4277-a100-bf3f85cadd92 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:20.987812 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00eb1b4d-a061-4277-a100-bf3f85cadd92 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:20.989869 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00eb1b4d-a061-4277-a100-bf3f85cadd92 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:20.993044 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00eb1b4d-a061-4277-a100-bf3f85cadd92 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:21.006765 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 948/1895] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:20 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:53:21.008427 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29d464ba-33f9-426e-8099-80b5b2ff81a5 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:21.012787 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 948/1896] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:53:20 2022] GET /identity/v3/roles => generated 1420 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:53:21.014099 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00eb1b4d-a061-4277-a100-bf3f85cadd92 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:21.026183 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00eb1b4d-a061-4277-a100-bf3f85cadd92 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:21.031689 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2ba3ae-63d8-4f58-adfa-de4286ab6bfe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:21.032470 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2ba3ae-63d8-4f58-adfa-de4286ab6bfe admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:21.032634 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2ba3ae-63d8-4f58-adfa-de4286ab6bfe admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:21.032904 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2ba3ae-63d8-4f58-adfa-de4286ab6bfe admin admin] PATH_INFO: `/v3/projects/12d8edb251ec4b20a3a383572f12fc52/users/2f87d0a253674c86b093ffcb590e07b1/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:21.056007 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf2ba3ae-63d8-4f58-adfa-de4286ab6bfe admin admin] RBAC: Authorizing `identity:create_grant(project_id=12d8edb251ec4b20a3a383572f12fc52, user_id=2f87d0a253674c86b093ffcb590e07b1, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:21.059224 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf2ba3ae-63d8-4f58-adfa-de4286ab6bfe admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:21.092756 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 949/1897] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:53:21 2022] PUT /identity/v3/projects/12d8edb251ec4b20a3a383572f12fc52/users/2f87d0a253674c86b093ffcb590e07b1/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 79 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:21.093832 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2ba3ae-63d8-4f58-adfa-de4286ab6bfe admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:21.098944 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-cf2ba3ae-63d8-4f58-adfa-de4286ab6bfe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:21.105495 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:21.114953 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29d464ba-33f9-426e-8099-80b5b2ff81a5 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:21.176658 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2ba3ae-63d8-4f58-adfa-de4286ab6bfe admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:21.180646 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a658e1-3690-4c91-bc6e-ac4957222199 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:21.181254 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a658e1-3690-4c91-bc6e-ac4957222199 service cinder] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:21.181533 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a658e1-3690-4c91-bc6e-ac4957222199 service cinder] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:21.181878 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a658e1-3690-4c91-bc6e-ac4957222199 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:21.207449 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c955923b-ba6d-4797-b8a2-1b1c5cafa388 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:21.208129 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c955923b-ba6d-4797-b8a2-1b1c5cafa388 service cinder] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:21.208346 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c955923b-ba6d-4797-b8a2-1b1c5cafa388 service cinder] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:21.208617 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c955923b-ba6d-4797-b8a2-1b1c5cafa388 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:21.210595 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-c955923b-ba6d-4797-b8a2-1b1c5cafa388 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:21.262572 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a658e1-3690-4c91-bc6e-ac4957222199 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:21.265486 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a658e1-3690-4c91-bc6e-ac4957222199 service cinder] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:21.274961 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c955923b-ba6d-4797-b8a2-1b1c5cafa388 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:21.276802 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c955923b-ba6d-4797-b8a2-1b1c5cafa388 service cinder] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:21.280726 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 950/1898] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:53:21 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 187 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:53:21.283329 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d1e6d4e-c24d-46b4-864d-202f0a081154 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:21.283711 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d1e6d4e-c24d-46b4-864d-202f0a081154 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:21.284295 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d1e6d4e-c24d-46b4-864d-202f0a081154 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:21.289929 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 949/1899] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:53:21 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 282 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:53:21.375291 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-0d1e6d4e-c24d-46b4-864d-202f0a081154 None None] MFA Rules not processed for user `2f87d0a253674c86b093ffcb590e07b1`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:21.376530 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-0d1e6d4e-c24d-46b4-864d-202f0a081154 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:21.496108 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c955923b-ba6d-4797-b8a2-1b1c5cafa388 service cinder] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:21.509036 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c955923b-ba6d-4797-b8a2-1b1c5cafa388 service cinder] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:21.513563 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-611acb93-6e6a-4a89-afe8-d1b3c68e2069 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:21.514399 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611acb93-6e6a-4a89-afe8-d1b3c68e2069 service cinder] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:21.514777 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611acb93-6e6a-4a89-afe8-d1b3c68e2069 service cinder] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:21.515208 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611acb93-6e6a-4a89-afe8-d1b3c68e2069 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:21.520107 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 951/1900] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:21 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 239 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:53:21.523295 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d1e6d4e-c24d-46b4-864d-202f0a081154 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:21.541655 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:21.601166 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-611acb93-6e6a-4a89-afe8-d1b3c68e2069 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:21.602810 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-611acb93-6e6a-4a89-afe8-d1b3c68e2069 service cinder] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:21.613101 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 950/1901] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:53:21 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 11 00:53:21.626089 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d1e6d4e-c24d-46b4-864d-202f0a081154 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:21.631180 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07135f0e-76ab-4acc-bae8-ddd58dfa6a89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:21.631917 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07135f0e-76ab-4acc-bae8-ddd58dfa6a89 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:21.632094 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07135f0e-76ab-4acc-bae8-ddd58dfa6a89 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:21.632365 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07135f0e-76ab-4acc-bae8-ddd58dfa6a89 admin admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:21.633003 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07135f0e-76ab-4acc-bae8-ddd58dfa6a89 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:21.634990 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07135f0e-76ab-4acc-bae8-ddd58dfa6a89 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:21.665563 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 952/1902] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:53:21 2022] POST /identity/v3/projects => generated 377 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:53:21.669899 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-611acb93-6e6a-4a89-afe8-d1b3c68e2069 service cinder] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:21.689442 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-611acb93-6e6a-4a89-afe8-d1b3c68e2069 service cinder] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:21.772529 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:21.790288 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b482e8-c328-4e6b-9fc2-bb2547df367a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:21.792278 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b482e8-c328-4e6b-9fc2-bb2547df367a admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:21.793197 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b482e8-c328-4e6b-9fc2-bb2547df367a admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:21.794781 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b482e8-c328-4e6b-9fc2-bb2547df367a admin admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:21.797122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8b482e8-c328-4e6b-9fc2-bb2547df367a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:21.800795 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8b482e8-c328-4e6b-9fc2-bb2547df367a admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:21.846212 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 951/1903] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:53:21 2022] POST /identity/v3/users => generated 295 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:53:21.850279 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07135f0e-76ab-4acc-bae8-ddd58dfa6a89 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:21.866337 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07135f0e-76ab-4acc-bae8-ddd58dfa6a89 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:21.876237 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06c31c4f-59ff-4ece-b7fc-e0b1907e9287 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:21.876866 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c31c4f-59ff-4ece-b7fc-e0b1907e9287 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:21.877141 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c31c4f-59ff-4ece-b7fc-e0b1907e9287 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:21.881361 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c31c4f-59ff-4ece-b7fc-e0b1907e9287 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:21.882197 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06c31c4f-59ff-4ece-b7fc-e0b1907e9287 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:21.887098 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06c31c4f-59ff-4ece-b7fc-e0b1907e9287 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:21.912723 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 953/1904] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:53:21 2022] GET /identity/v3/roles => generated 1420 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:53:21.914506 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b482e8-c328-4e6b-9fc2-bb2547df367a admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:21.930741 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b482e8-c328-4e6b-9fc2-bb2547df367a admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:21.944931 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c9d9f5b-85c6-4187-a400-7b8671146a45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:21.946074 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9d9f5b-85c6-4187-a400-7b8671146a45 admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:21.946455 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9d9f5b-85c6-4187-a400-7b8671146a45 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:21.947415 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9d9f5b-85c6-4187-a400-7b8671146a45 admin admin] PATH_INFO: `/v3/projects/d297158cd9e341e4a094266fef68fa07/users/15362a7e5cba4d6f86ded3d91654239f/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:21.989349 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c9d9f5b-85c6-4187-a400-7b8671146a45 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d297158cd9e341e4a094266fef68fa07, user_id=15362a7e5cba4d6f86ded3d91654239f, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:21.993432 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c9d9f5b-85c6-4187-a400-7b8671146a45 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:22.042549 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 952/1905] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:53:21 2022] PUT /identity/v3/projects/d297158cd9e341e4a094266fef68fa07/users/15362a7e5cba4d6f86ded3d91654239f/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:22.046223 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06c31c4f-59ff-4ece-b7fc-e0b1907e9287 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:22.080855 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06c31c4f-59ff-4ece-b7fc-e0b1907e9287 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:22.224659 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d195f034-d585-4fb9-b462-0d08fcca259b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:22.226104 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d195f034-d585-4fb9-b462-0d08fcca259b admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:22.226327 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d195f034-d585-4fb9-b462-0d08fcca259b admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:22.226599 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d195f034-d585-4fb9-b462-0d08fcca259b admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:22.227347 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d195f034-d585-4fb9-b462-0d08fcca259b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:22.229701 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d195f034-d585-4fb9-b462-0d08fcca259b admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:22.244904 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 954/1906] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:53:22 2022] GET /identity/v3/roles => generated 1420 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:53:22.248958 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c9d9f5b-85c6-4187-a400-7b8671146a45 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:22.261196 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c9d9f5b-85c6-4187-a400-7b8671146a45 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:22.267247 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca41ebf8-4535-4025-af20-0fbe749c77bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:22.267917 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca41ebf8-4535-4025-af20-0fbe749c77bd admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:22.268196 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca41ebf8-4535-4025-af20-0fbe749c77bd admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:22.268462 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca41ebf8-4535-4025-af20-0fbe749c77bd admin admin] PATH_INFO: `/v3/projects/d297158cd9e341e4a094266fef68fa07/users/15362a7e5cba4d6f86ded3d91654239f/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:22.270379 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca41ebf8-4535-4025-af20-0fbe749c77bd admin admin] RBAC: Authorizing `identity:create_grant(project_id=d297158cd9e341e4a094266fef68fa07, user_id=15362a7e5cba4d6f86ded3d91654239f, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:22.273060 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca41ebf8-4535-4025-af20-0fbe749c77bd admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:22.286522 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 953/1907] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:53:22 2022] PUT /identity/v3/projects/d297158cd9e341e4a094266fef68fa07/users/15362a7e5cba4d6f86ded3d91654239f/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:53:22.297239 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd729462-85fb-4674-9506-6086a3f74592 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:22.297524 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd729462-85fb-4674-9506-6086a3f74592 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:22.297800 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd729462-85fb-4674-9506-6086a3f74592 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:22.329776 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-fd729462-85fb-4674-9506-6086a3f74592 None None] MFA Rules not processed for user `15362a7e5cba4d6f86ded3d91654239f`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:22.330833 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-fd729462-85fb-4674-9506-6086a3f74592 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:22.439117 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:22.464859 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 955/1908] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:22 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:53:22.479412 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca41ebf8-4535-4025-af20-0fbe749c77bd admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:22.577110 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca41ebf8-4535-4025-af20-0fbe749c77bd admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:22.582948 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-455a1356-f440-4d16-8813-ca87ef737f2b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:22.584775 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455a1356-f440-4d16-8813-ca87ef737f2b service cinder] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:22.585144 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455a1356-f440-4d16-8813-ca87ef737f2b service cinder] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:22.585541 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455a1356-f440-4d16-8813-ca87ef737f2b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:22.720296 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-455a1356-f440-4d16-8813-ca87ef737f2b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:22.721764 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-455a1356-f440-4d16-8813-ca87ef737f2b service cinder] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:22.749613 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 954/1909] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:53:22 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 271 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:53:24.771757 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd729462-85fb-4674-9506-6086a3f74592 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:24.876107 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd729462-85fb-4674-9506-6086a3f74592 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:24.881398 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c91414d-d0bb-4879-a1d9-d4d5a495ed57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:24.883886 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c91414d-d0bb-4879-a1d9-d4d5a495ed57 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:24.884409 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c91414d-d0bb-4879-a1d9-d4d5a495ed57 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:24.885322 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c91414d-d0bb-4879-a1d9-d4d5a495ed57 admin admin] PATH_INFO: `/v3/users/de3d4478bbb1418dbbb8c048a650d8c4` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:24.889824 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c91414d-d0bb-4879-a1d9-d4d5a495ed57 admin admin] RBAC: Authorizing `identity:delete_user(user_id=de3d4478bbb1418dbbb8c048a650d8c4)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:24.892463 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c91414d-d0bb-4879-a1d9-d4d5a495ed57 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:24.969287 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-4c91414d-d0bb-4879-a1d9-d4d5a495ed57 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'de3d4478bbb1418dbbb8c048a650d8c4'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:24.970024 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-4c91414d-d0bb-4879-a1d9-d4d5a495ed57 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'de3d4478bbb1418dbbb8c048a650d8c4'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:24.970817 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-4c91414d-d0bb-4879-a1d9-d4d5a495ed57 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'de3d4478bbb1418dbbb8c048a650d8c4'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:24.981919 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-4c91414d-d0bb-4879-a1d9-d4d5a495ed57 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'de3d4478bbb1418dbbb8c048a650d8c4'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:25.003385 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-4c91414d-d0bb-4879-a1d9-d4d5a495ed57 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'de3d4478bbb1418dbbb8c048a650d8c4'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:25.007437 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-455a1356-f440-4d16-8813-ca87ef737f2b service cinder] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:25.009500 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-455a1356-f440-4d16-8813-ca87ef737f2b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:25.027493 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 956/1910] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:53:24 2022] DELETE /identity/v3/users/de3d4478bbb1418dbbb8c048a650d8c4 => generated 0 bytes in 256 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:53:25.032149 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db50bedf-655e-4e3f-85cc-a860efd4c51a None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:25.032544 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db50bedf-655e-4e3f-85cc-a860efd4c51a None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:25.032917 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db50bedf-655e-4e3f-85cc-a860efd4c51a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:25.036296 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-455a1356-f440-4d16-8813-ca87ef737f2b service cinder] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:25.053678 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-db50bedf-655e-4e3f-85cc-a860efd4c51a None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:25.055084 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-db50bedf-655e-4e3f-85cc-a860efd4c51a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:25.101471 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:25.111508 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb168fc-289a-4696-9d61-7eb5e4d0a917 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:25.112155 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb168fc-289a-4696-9d61-7eb5e4d0a917 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:25.112493 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb168fc-289a-4696-9d61-7eb5e4d0a917 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:25.112964 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb168fc-289a-4696-9d61-7eb5e4d0a917 admin admin] PATH_INFO: `/v3/users/9048531ea3c64fe793aec2de9b56b2bf` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:25.114376 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eb168fc-289a-4696-9d61-7eb5e4d0a917 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9048531ea3c64fe793aec2de9b56b2bf)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:25.116880 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eb168fc-289a-4696-9d61-7eb5e4d0a917 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:25.142740 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 957/1911] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:53:25.154109 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db50bedf-655e-4e3f-85cc-a860efd4c51a None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:25.156036 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-db50bedf-655e-4e3f-85cc-a860efd4c51a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:25.186291 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:25.209067 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-2eb168fc-289a-4696-9d61-7eb5e4d0a917 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9048531ea3c64fe793aec2de9b56b2bf'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:25.210102 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-2eb168fc-289a-4696-9d61-7eb5e4d0a917 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9048531ea3c64fe793aec2de9b56b2bf'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:25.211330 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-2eb168fc-289a-4696-9d61-7eb5e4d0a917 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9048531ea3c64fe793aec2de9b56b2bf'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:25.222983 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-2eb168fc-289a-4696-9d61-7eb5e4d0a917 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9048531ea3c64fe793aec2de9b56b2bf'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:25.230908 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db50bedf-655e-4e3f-85cc-a860efd4c51a None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:25.239350 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d88d8df6-564b-4f73-a9f1-e3f339ed505f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:25.239979 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88d8df6-564b-4f73-a9f1-e3f339ed505f service neutron] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:25.240258 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88d8df6-564b-4f73-a9f1-e3f339ed505f service neutron] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:25.240532 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88d8df6-564b-4f73-a9f1-e3f339ed505f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:25.242547 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-d88d8df6-564b-4f73-a9f1-e3f339ed505f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:25.243295 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-2eb168fc-289a-4696-9d61-7eb5e4d0a917 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9048531ea3c64fe793aec2de9b56b2bf'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:25.255524 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d88d8df6-564b-4f73-a9f1-e3f339ed505f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:25.256984 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d88d8df6-564b-4f73-a9f1-e3f339ed505f service neutron] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:25.261862 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 955/1912] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:53:25 2022] DELETE /identity/v3/users/9048531ea3c64fe793aec2de9b56b2bf => generated 0 bytes in 255 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:53:25.273869 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 958/1913] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:53:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 11 00:53:25.600685 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb168fc-289a-4696-9d61-7eb5e4d0a917 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:25.601377 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-2eb168fc-289a-4696-9d61-7eb5e4d0a917 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:25.691177 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb168fc-289a-4696-9d61-7eb5e4d0a917 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:25.695887 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56765b07-1e3a-4cbe-adb7-72d76a267ecb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:25.696525 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56765b07-1e3a-4cbe-adb7-72d76a267ecb admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:25.696797 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56765b07-1e3a-4cbe-adb7-72d76a267ecb admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:25.697064 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56765b07-1e3a-4cbe-adb7-72d76a267ecb admin admin] PATH_INFO: `/v3/projects/47891decc6d14bffb70bb4fb7c46c458` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:25.698195 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56765b07-1e3a-4cbe-adb7-72d76a267ecb admin admin] RBAC: Authorizing `identity:delete_project(project_id=47891decc6d14bffb70bb4fb7c46c458)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:25.700227 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56765b07-1e3a-4cbe-adb7-72d76a267ecb admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:25.750373 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d88d8df6-564b-4f73-a9f1-e3f339ed505f service neutron] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:25.752248 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-d88d8df6-564b-4f73-a9f1-e3f339ed505f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:25.765251 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-56765b07-1e3a-4cbe-adb7-72d76a267ecb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '47891decc6d14bffb70bb4fb7c46c458', 'request_id': 'req-56765b07-1e3a-4cbe-adb7-72d76a267ecb'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:25.767154 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d88d8df6-564b-4f73-a9f1-e3f339ed505f service neutron] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:25.772318 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-56765b07-1e3a-4cbe-adb7-72d76a267ecb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '47891decc6d14bffb70bb4fb7c46c458', 'request_id': 'req-56765b07-1e3a-4cbe-adb7-72d76a267ecb'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:25.782561 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-56765b07-1e3a-4cbe-adb7-72d76a267ecb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '47891decc6d14bffb70bb4fb7c46c458', 'request_id': 'req-56765b07-1e3a-4cbe-adb7-72d76a267ecb'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:25.795194 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-56765b07-1e3a-4cbe-adb7-72d76a267ecb admin admin] The token cache is being invalidate because project 47891decc6d14bffb70bb4fb7c46c458 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87924) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:53:25.795480 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-56765b07-1e3a-4cbe-adb7-72d76a267ecb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:25.796133 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-56765b07-1e3a-4cbe-adb7-72d76a267ecb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:25.797664 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 956/1914] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:53:25 2022] DELETE /identity/v3/projects/47891decc6d14bffb70bb4fb7c46c458 => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:25.801426 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56765b07-1e3a-4cbe-adb7-72d76a267ecb admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:25.803221 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-56765b07-1e3a-4cbe-adb7-72d76a267ecb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:25.812920 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:25.829780 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87478527-bfdb-492f-8fd3-308d4fd2c08c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:25.830416 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87478527-bfdb-492f-8fd3-308d4fd2c08c admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:25.830691 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87478527-bfdb-492f-8fd3-308d4fd2c08c admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:25.830991 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87478527-bfdb-492f-8fd3-308d4fd2c08c admin admin] PATH_INFO: `/v3/projects/afbed9f160d8407598fa6257b82c9c28` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:25.832120 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87478527-bfdb-492f-8fd3-308d4fd2c08c admin admin] RBAC: Authorizing `identity:delete_project(project_id=afbed9f160d8407598fa6257b82c9c28)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:25.834036 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87478527-bfdb-492f-8fd3-308d4fd2c08c admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:25.873236 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56765b07-1e3a-4cbe-adb7-72d76a267ecb admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:25.880611 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-87478527-bfdb-492f-8fd3-308d4fd2c08c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'afbed9f160d8407598fa6257b82c9c28', 'request_id': 'req-87478527-bfdb-492f-8fd3-308d4fd2c08c'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:25.886811 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-87478527-bfdb-492f-8fd3-308d4fd2c08c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'afbed9f160d8407598fa6257b82c9c28', 'request_id': 'req-87478527-bfdb-492f-8fd3-308d4fd2c08c'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:25.892918 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-87478527-bfdb-492f-8fd3-308d4fd2c08c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'afbed9f160d8407598fa6257b82c9c28', 'request_id': 'req-87478527-bfdb-492f-8fd3-308d4fd2c08c'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:25.906342 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-87478527-bfdb-492f-8fd3-308d4fd2c08c admin admin] The token cache is being invalidate because project afbed9f160d8407598fa6257b82c9c28 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87923) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:53:25.906629 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-87478527-bfdb-492f-8fd3-308d4fd2c08c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:25.907323 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-87478527-bfdb-492f-8fd3-308d4fd2c08c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:25.909384 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 959/1915] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:53:25 2022] DELETE /identity/v3/projects/afbed9f160d8407598fa6257b82c9c28 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:25.913251 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87478527-bfdb-492f-8fd3-308d4fd2c08c admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:25.914547 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-87478527-bfdb-492f-8fd3-308d4fd2c08c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:25.933821 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b22326a-a926-4ec2-907c-318a03ad9f4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:25.934493 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b22326a-a926-4ec2-907c-318a03ad9f4c admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:25.934780 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b22326a-a926-4ec2-907c-318a03ad9f4c admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:25.935082 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b22326a-a926-4ec2-907c-318a03ad9f4c admin admin] PATH_INFO: `/v3/users/235c65cabb824f27b4ee5c49f6b6f878` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:25.936229 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b22326a-a926-4ec2-907c-318a03ad9f4c admin admin] RBAC: Authorizing `identity:delete_user(user_id=235c65cabb824f27b4ee5c49f6b6f878)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:25.938505 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b22326a-a926-4ec2-907c-318a03ad9f4c admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:25.972555 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:25.987428 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87478527-bfdb-492f-8fd3-308d4fd2c08c admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:25.992786 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25021243-a8cf-4d26-96a0-11e0c2f0b0f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:25.993431 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25021243-a8cf-4d26-96a0-11e0c2f0b0f3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:25.993764 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25021243-a8cf-4d26-96a0-11e0c2f0b0f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:25.994040 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25021243-a8cf-4d26-96a0-11e0c2f0b0f3 admin admin] PATH_INFO: `/v3/users/b0409468f7a140ca9b7ab2cba292c84a` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:25.995391 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25021243-a8cf-4d26-96a0-11e0c2f0b0f3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b0409468f7a140ca9b7ab2cba292c84a)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:25.996173 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-0b22326a-a926-4ec2-907c-318a03ad9f4c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '235c65cabb824f27b4ee5c49f6b6f878'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:25.998065 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-0b22326a-a926-4ec2-907c-318a03ad9f4c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '235c65cabb824f27b4ee5c49f6b6f878'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:25.998181 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25021243-a8cf-4d26-96a0-11e0c2f0b0f3 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:25.999148 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-0b22326a-a926-4ec2-907c-318a03ad9f4c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '235c65cabb824f27b4ee5c49f6b6f878'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:26.007349 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-0b22326a-a926-4ec2-907c-318a03ad9f4c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '235c65cabb824f27b4ee5c49f6b6f878'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:26.022120 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-0b22326a-a926-4ec2-907c-318a03ad9f4c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '235c65cabb824f27b4ee5c49f6b6f878'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:26.040844 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 957/1916] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:53:25 2022] DELETE /identity/v3/users/235c65cabb824f27b4ee5c49f6b6f878 => generated 0 bytes in 240 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:53:26.058443 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-25021243-a8cf-4d26-96a0-11e0c2f0b0f3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b0409468f7a140ca9b7ab2cba292c84a'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:26.059189 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-25021243-a8cf-4d26-96a0-11e0c2f0b0f3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b0409468f7a140ca9b7ab2cba292c84a'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:26.059886 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-25021243-a8cf-4d26-96a0-11e0c2f0b0f3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b0409468f7a140ca9b7ab2cba292c84a'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:26.068754 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-25021243-a8cf-4d26-96a0-11e0c2f0b0f3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b0409468f7a140ca9b7ab2cba292c84a'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:26.085125 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-25021243-a8cf-4d26-96a0-11e0c2f0b0f3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b0409468f7a140ca9b7ab2cba292c84a'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:26.100976 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 960/1917] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:53:25 2022] DELETE /identity/v3/users/b0409468f7a140ca9b7ab2cba292c84a => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:53:26.347996 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b22326a-a926-4ec2-907c-318a03ad9f4c admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:26.350149 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-0b22326a-a926-4ec2-907c-318a03ad9f4c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:26.416562 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:26.449991 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b22326a-a926-4ec2-907c-318a03ad9f4c admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:26.508896 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f313b35-fd11-4f96-892c-8e30dd145971 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:26.509544 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f313b35-fd11-4f96-892c-8e30dd145971 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:26.509972 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f313b35-fd11-4f96-892c-8e30dd145971 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:26.510254 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f313b35-fd11-4f96-892c-8e30dd145971 admin admin] PATH_INFO: `/v3/projects/f8c5ffd6786c4a3f9bd9462db458af33` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:26.511621 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f313b35-fd11-4f96-892c-8e30dd145971 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f8c5ffd6786c4a3f9bd9462db458af33)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:26.514164 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f313b35-fd11-4f96-892c-8e30dd145971 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:26.537224 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25021243-a8cf-4d26-96a0-11e0c2f0b0f3 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:26.538696 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-25021243-a8cf-4d26-96a0-11e0c2f0b0f3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:26.556501 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25021243-a8cf-4d26-96a0-11e0c2f0b0f3 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:26.570893 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290e1b7d-f400-4afc-a58b-f45886d305d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:26.571593 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290e1b7d-f400-4afc-a58b-f45886d305d9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:26.571977 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290e1b7d-f400-4afc-a58b-f45886d305d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:26.572254 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290e1b7d-f400-4afc-a58b-f45886d305d9 admin admin] PATH_INFO: `/v3/projects/4c08a4f7d5f844e8af87c9c0d22cdfa3` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:26.573456 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-290e1b7d-f400-4afc-a58b-f45886d305d9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4c08a4f7d5f844e8af87c9c0d22cdfa3)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:26.576050 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-290e1b7d-f400-4afc-a58b-f45886d305d9 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:26.579930 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-5f313b35-fd11-4f96-892c-8e30dd145971 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f8c5ffd6786c4a3f9bd9462db458af33', 'request_id': 'req-5f313b35-fd11-4f96-892c-8e30dd145971'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:26.587010 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-5f313b35-fd11-4f96-892c-8e30dd145971 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f8c5ffd6786c4a3f9bd9462db458af33', 'request_id': 'req-5f313b35-fd11-4f96-892c-8e30dd145971'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:26.596334 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-5f313b35-fd11-4f96-892c-8e30dd145971 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f8c5ffd6786c4a3f9bd9462db458af33', 'request_id': 'req-5f313b35-fd11-4f96-892c-8e30dd145971'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:26.612703 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-5f313b35-fd11-4f96-892c-8e30dd145971 admin admin] The token cache is being invalidate because project f8c5ffd6786c4a3f9bd9462db458af33 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87924) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:53:26.613588 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-5f313b35-fd11-4f96-892c-8e30dd145971 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:26.614388 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-5f313b35-fd11-4f96-892c-8e30dd145971 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:26.616210 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 958/1918] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:53:26 2022] DELETE /identity/v3/projects/f8c5ffd6786c4a3f9bd9462db458af33 => generated 0 bytes in 268 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:26.621197 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-531fd6f6-f627-4c07-a8cb-3e2f6600baea None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:26.621513 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-531fd6f6-f627-4c07-a8cb-3e2f6600baea None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:26.621800 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-531fd6f6-f627-4c07-a8cb-3e2f6600baea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:26.632034 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-290e1b7d-f400-4afc-a58b-f45886d305d9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4c08a4f7d5f844e8af87c9c0d22cdfa3', 'request_id': 'req-290e1b7d-f400-4afc-a58b-f45886d305d9'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:26.639429 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-290e1b7d-f400-4afc-a58b-f45886d305d9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4c08a4f7d5f844e8af87c9c0d22cdfa3', 'request_id': 'req-290e1b7d-f400-4afc-a58b-f45886d305d9'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:26.646088 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-531fd6f6-f627-4c07-a8cb-3e2f6600baea None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:26.647261 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-290e1b7d-f400-4afc-a58b-f45886d305d9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4c08a4f7d5f844e8af87c9c0d22cdfa3', 'request_id': 'req-290e1b7d-f400-4afc-a58b-f45886d305d9'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:26.647524 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-531fd6f6-f627-4c07-a8cb-3e2f6600baea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:26.664598 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-290e1b7d-f400-4afc-a58b-f45886d305d9 admin admin] The token cache is being invalidate because project 4c08a4f7d5f844e8af87c9c0d22cdfa3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87923) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:53:26.665266 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-290e1b7d-f400-4afc-a58b-f45886d305d9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:26.665549 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-290e1b7d-f400-4afc-a58b-f45886d305d9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:26.670884 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 961/1919] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:53:26 2022] DELETE /identity/v3/projects/4c08a4f7d5f844e8af87c9c0d22cdfa3 => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:26.673365 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563b15b4-4615-4f90-b1b8-6fa8f16f8d0a None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:26.673679 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563b15b4-4615-4f90-b1b8-6fa8f16f8d0a None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:26.673926 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563b15b4-4615-4f90-b1b8-6fa8f16f8d0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:26.694335 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-563b15b4-4615-4f90-b1b8-6fa8f16f8d0a None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:26.695581 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-563b15b4-4615-4f90-b1b8-6fa8f16f8d0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:26.722571 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 959/1920] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:53:26.730289 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72433fcd-94af-44d4-aa5a-253cc867f01d None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:26.730578 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72433fcd-94af-44d4-aa5a-253cc867f01d None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:26.730892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72433fcd-94af-44d4-aa5a-253cc867f01d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:26.751742 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:26.753164 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-72433fcd-94af-44d4-aa5a-253cc867f01d None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:26.754505 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-72433fcd-94af-44d4-aa5a-253cc867f01d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:26.765858 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 960/1921] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:53:26.769937 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72433fcd-94af-44d4-aa5a-253cc867f01d None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:26.783030 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:26.784077 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 962/1922] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:53:26.787658 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72433fcd-94af-44d4-aa5a-253cc867f01d None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:26.792712 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2601096-8a4f-41b8-82cd-4734b5ea40de None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:26.793001 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2601096-8a4f-41b8-82cd-4734b5ea40de None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:26.793464 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2601096-8a4f-41b8-82cd-4734b5ea40de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:26.794994 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb33326-10c8-46da-8dac-bab50c368838 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:26.795804 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb33326-10c8-46da-8dac-bab50c368838 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:26.796547 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb33326-10c8-46da-8dac-bab50c368838 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:26.797006 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb33326-10c8-46da-8dac-bab50c368838 admin admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:26.797752 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cb33326-10c8-46da-8dac-bab50c368838 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:53:26.798156 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cb33326-10c8-46da-8dac-bab50c368838 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:26.800227 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cb33326-10c8-46da-8dac-bab50c368838 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:26.812497 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 961/1923] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 00:53:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:53:26.816636 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb33326-10c8-46da-8dac-bab50c368838 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:26.816769 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-e2601096-8a4f-41b8-82cd-4734b5ea40de None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:26.817875 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-e2601096-8a4f-41b8-82cd-4734b5ea40de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:26.826865 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 963/1924] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:53:26.827159 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb33326-10c8-46da-8dac-bab50c368838 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:26.830947 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2601096-8a4f-41b8-82cd-4734b5ea40de None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:26.831995 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5659cde6-bce1-46a9-bdfa-72c01ef26bc2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:26.832580 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5659cde6-bce1-46a9-bdfa-72c01ef26bc2 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:26.832853 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5659cde6-bce1-46a9-bdfa-72c01ef26bc2 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:26.833136 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5659cde6-bce1-46a9-bdfa-72c01ef26bc2 admin admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:26.833736 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5659cde6-bce1-46a9-bdfa-72c01ef26bc2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:26.835655 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5659cde6-bce1-46a9-bdfa-72c01ef26bc2 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:26.840239 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2601096-8a4f-41b8-82cd-4734b5ea40de None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:26.845528 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f5e0200-d2fb-4383-9702-edc74964f5ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:26.846125 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5e0200-d2fb-4383-9702-edc74964f5ba admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:26.846402 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5e0200-d2fb-4383-9702-edc74964f5ba admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:26.846708 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5e0200-d2fb-4383-9702-edc74964f5ba admin admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:26.847390 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f5e0200-d2fb-4383-9702-edc74964f5ba admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:53:26.847691 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f5e0200-d2fb-4383-9702-edc74964f5ba admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:26.849276 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f5e0200-d2fb-4383-9702-edc74964f5ba admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:26.861255 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 964/1925] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 00:53:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:53:26.863762 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 962/1926] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:53:26 2022] POST /identity/v3/projects => generated 375 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:53:26.865609 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f5e0200-d2fb-4383-9702-edc74964f5ba admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:26.867567 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5659cde6-bce1-46a9-bdfa-72c01ef26bc2 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:26.875559 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f5e0200-d2fb-4383-9702-edc74964f5ba admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:26.928931 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5659cde6-bce1-46a9-bdfa-72c01ef26bc2 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:26.933442 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845f8719-bceb-4ffa-a660-643d9b3af05d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:26.934040 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845f8719-bceb-4ffa-a660-643d9b3af05d admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:26.934320 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845f8719-bceb-4ffa-a660-643d9b3af05d admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:26.934682 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845f8719-bceb-4ffa-a660-643d9b3af05d admin admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:26.935076 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09457bb2-4e51-4ead-9d95-69d2cbb1ed2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:26.935308 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-845f8719-bceb-4ffa-a660-643d9b3af05d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:26.935678 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09457bb2-4e51-4ead-9d95-69d2cbb1ed2f admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:26.935959 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09457bb2-4e51-4ead-9d95-69d2cbb1ed2f admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:26.936232 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09457bb2-4e51-4ead-9d95-69d2cbb1ed2f admin admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:26.936863 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09457bb2-4e51-4ead-9d95-69d2cbb1ed2f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:26.937218 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-845f8719-bceb-4ffa-a660-643d9b3af05d admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:26.938758 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09457bb2-4e51-4ead-9d95-69d2cbb1ed2f admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:26.959596 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 963/1927] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:53:26 2022] POST /identity/v3/users => generated 288 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:53:26.963853 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845f8719-bceb-4ffa-a660-643d9b3af05d admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:26.968259 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 965/1928] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:53:26 2022] POST /identity/v3/projects => generated 369 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:53:26.972491 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09457bb2-4e51-4ead-9d95-69d2cbb1ed2f admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:26.975221 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845f8719-bceb-4ffa-a660-643d9b3af05d admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:27.033423 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bae28cb-c39c-401e-bec2-94a1f3703e2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:27.034150 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bae28cb-c39c-401e-bec2-94a1f3703e2e admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:27.034270 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bae28cb-c39c-401e-bec2-94a1f3703e2e admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:27.034544 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bae28cb-c39c-401e-bec2-94a1f3703e2e admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:27.035485 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bae28cb-c39c-401e-bec2-94a1f3703e2e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:27.037100 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bae28cb-c39c-401e-bec2-94a1f3703e2e admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:27.038443 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09457bb2-4e51-4ead-9d95-69d2cbb1ed2f admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:27.043249 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e7b36ff-24b6-409a-bb1a-f91fb6b7a025 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:27.043859 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7b36ff-24b6-409a-bb1a-f91fb6b7a025 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:27.044142 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7b36ff-24b6-409a-bb1a-f91fb6b7a025 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:27.044425 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7b36ff-24b6-409a-bb1a-f91fb6b7a025 admin admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:27.045085 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e7b36ff-24b6-409a-bb1a-f91fb6b7a025 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:27.047059 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e7b36ff-24b6-409a-bb1a-f91fb6b7a025 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:27.047490 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 964/1929] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:53:26 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 11 00:53:27.051271 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bae28cb-c39c-401e-bec2-94a1f3703e2e admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:27.061695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bae28cb-c39c-401e-bec2-94a1f3703e2e admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:27.067482 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e034592e-eda5-428d-af32-90267a8c770c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:27.068169 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e034592e-eda5-428d-af32-90267a8c770c admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:27.068611 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e034592e-eda5-428d-af32-90267a8c770c admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:27.068952 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e034592e-eda5-428d-af32-90267a8c770c admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:27.069597 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e034592e-eda5-428d-af32-90267a8c770c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:27.071225 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e034592e-eda5-428d-af32-90267a8c770c admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:27.075429 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 966/1930] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:53:26 2022] POST /identity/v3/users => generated 285 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:53:27.076244 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e7b36ff-24b6-409a-bb1a-f91fb6b7a025 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:27.082222 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 965/1931] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:53:27 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 11 00:53:27.086461 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e034592e-eda5-428d-af32-90267a8c770c admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:27.092664 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e7b36ff-24b6-409a-bb1a-f91fb6b7a025 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:27.096690 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e034592e-eda5-428d-af32-90267a8c770c admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:27.097346 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edc10893-7ce0-4698-84dd-41690ce2ec8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:27.097972 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc10893-7ce0-4698-84dd-41690ce2ec8b admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:27.098250 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc10893-7ce0-4698-84dd-41690ce2ec8b admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:27.098522 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc10893-7ce0-4698-84dd-41690ce2ec8b admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:27.099225 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edc10893-7ce0-4698-84dd-41690ce2ec8b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:27.100822 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edc10893-7ce0-4698-84dd-41690ce2ec8b admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:27.101590 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a0c69ac-bc7a-4a2a-809e-26c0260312a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:27.102243 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0c69ac-bc7a-4a2a-809e-26c0260312a4 admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:27.102523 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0c69ac-bc7a-4a2a-809e-26c0260312a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:27.102823 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0c69ac-bc7a-4a2a-809e-26c0260312a4 admin admin] PATH_INFO: `/v3/projects/7beb504f4bc7405589196a3a532f38c8/users/589b5f57ac6c4838ae117daf3aa2423d/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:27.112634 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 967/1932] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:53:27 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 11 00:53:27.116494 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edc10893-7ce0-4698-84dd-41690ce2ec8b admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:27.118445 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a0c69ac-bc7a-4a2a-809e-26c0260312a4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7beb504f4bc7405589196a3a532f38c8, user_id=589b5f57ac6c4838ae117daf3aa2423d, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:27.121134 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a0c69ac-bc7a-4a2a-809e-26c0260312a4 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:27.127096 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edc10893-7ce0-4698-84dd-41690ce2ec8b admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:27.132017 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0133fb5-7291-4c53-9d11-cbabc79cd03a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:27.132606 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0133fb5-7291-4c53-9d11-cbabc79cd03a admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:27.132880 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0133fb5-7291-4c53-9d11-cbabc79cd03a admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:27.133197 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0133fb5-7291-4c53-9d11-cbabc79cd03a admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:27.133828 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0133fb5-7291-4c53-9d11-cbabc79cd03a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:27.135438 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0133fb5-7291-4c53-9d11-cbabc79cd03a admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:27.135675 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 966/1933] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:53:27 2022] PUT /identity/v3/projects/7beb504f4bc7405589196a3a532f38c8/users/589b5f57ac6c4838ae117daf3aa2423d/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:27.145738 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf149c6-93d9-4d9d-bb09-2b9da0e1a7d4 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:27.146026 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf149c6-93d9-4d9d-bb09-2b9da0e1a7d4 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:27.146383 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 968/1934] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:53:27 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 11 00:53:27.146500 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf149c6-93d9-4d9d-bb09-2b9da0e1a7d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:27.150283 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0133fb5-7291-4c53-9d11-cbabc79cd03a admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:27.160882 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0133fb5-7291-4c53-9d11-cbabc79cd03a admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:27.176077 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-abf149c6-93d9-4d9d-bb09-2b9da0e1a7d4 None None] MFA Rules not processed for user `589b5f57ac6c4838ae117daf3aa2423d`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:27.177301 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-abf149c6-93d9-4d9d-bb09-2b9da0e1a7d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:27.190920 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:27.191628 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:27.229451 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6faf134-e4f5-4804-9b93-430c7a1fd931 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:27.230236 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6faf134-e4f5-4804-9b93-430c7a1fd931 admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:27.230899 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6faf134-e4f5-4804-9b93-430c7a1fd931 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:27.230899 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6faf134-e4f5-4804-9b93-430c7a1fd931 admin admin] PATH_INFO: `/v3/projects/733d0b2534874180a901f6b9eb352ff4/users/28d245fb98df47d889c0685e36cc5f22/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:27.249064 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6faf134-e4f5-4804-9b93-430c7a1fd931 admin admin] RBAC: Authorizing `identity:create_grant(project_id=733d0b2534874180a901f6b9eb352ff4, user_id=28d245fb98df47d889c0685e36cc5f22, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:27.252633 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6faf134-e4f5-4804-9b93-430c7a1fd931 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:27.257006 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 967/1935] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:27 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:53:27.266635 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 969/1936] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:53:27 2022] PUT /identity/v3/projects/733d0b2534874180a901f6b9eb352ff4/users/28d245fb98df47d889c0685e36cc5f22/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:53:27.277724 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf149c6-93d9-4d9d-bb09-2b9da0e1a7d4 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:27.278132 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74649b9e-2594-4913-9efb-9e6f41c51ed6 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:27.278967 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74649b9e-2594-4913-9efb-9e6f41c51ed6 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:27.279239 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74649b9e-2594-4913-9efb-9e6f41c51ed6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:27.281238 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-abf149c6-93d9-4d9d-bb09-2b9da0e1a7d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:27.310696 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-74649b9e-2594-4913-9efb-9e6f41c51ed6 None None] MFA Rules not processed for user `28d245fb98df47d889c0685e36cc5f22`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:27.312072 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-74649b9e-2594-4913-9efb-9e6f41c51ed6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:27.376393 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf149c6-93d9-4d9d-bb09-2b9da0e1a7d4 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:27.390937 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 970/1937] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:27 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:53:27.395474 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74649b9e-2594-4913-9efb-9e6f41c51ed6 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:27.425557 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e17bc41b-d982-403c-9b12-d44667df4753 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '06b1c911731f4267a5f37e91d948e7e9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:27.426167 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17bc41b-d982-403c-9b12-d44667df4753 service glance] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:27.426442 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17bc41b-d982-403c-9b12-d44667df4753 service glance] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:27.426708 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17bc41b-d982-403c-9b12-d44667df4753 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:27.459894 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74649b9e-2594-4913-9efb-9e6f41c51ed6 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:27.464568 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf69051-0df1-456c-8c3a-062812743e24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:27.465174 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf69051-0df1-456c-8c3a-062812743e24 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:27.465454 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf69051-0df1-456c-8c3a-062812743e24 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:27.465862 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf69051-0df1-456c-8c3a-062812743e24 admin admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:27.466528 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cf69051-0df1-456c-8c3a-062812743e24 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:27.468532 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cf69051-0df1-456c-8c3a-062812743e24 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:27.484952 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e17bc41b-d982-403c-9b12-d44667df4753 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:27.486502 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e17bc41b-d982-403c-9b12-d44667df4753 service glance] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:27.491599 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:27.499594 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 971/1938] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:53:27 2022] POST /identity/v3/projects => generated 367 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:53:27.499771 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 968/1939] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:53:27 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 220 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 11 00:53:27.500549 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf69051-0df1-456c-8c3a-062812743e24 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:27.532173 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:27.576403 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf69051-0df1-456c-8c3a-062812743e24 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:27.582656 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f72d6d4-4345-4fdb-a9f8-e69a4c80af59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:27.583481 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f72d6d4-4345-4fdb-a9f8-e69a4c80af59 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:27.583899 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f72d6d4-4345-4fdb-a9f8-e69a4c80af59 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:27.584326 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f72d6d4-4345-4fdb-a9f8-e69a4c80af59 admin admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:27.585392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f72d6d4-4345-4fdb-a9f8-e69a4c80af59 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:27.587922 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f72d6d4-4345-4fdb-a9f8-e69a4c80af59 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:27.619639 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 972/1940] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:53:27 2022] POST /identity/v3/users => generated 290 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:53:27.624694 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e17bc41b-d982-403c-9b12-d44667df4753 service glance] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:27.651191 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e17bc41b-d982-403c-9b12-d44667df4753 service glance] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:27.664191 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3e91765-38cf-4ca2-abcb-eb7d271b5621 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:27.664386 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e91765-38cf-4ca2-abcb-eb7d271b5621 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:27.664675 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e91765-38cf-4ca2-abcb-eb7d271b5621 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:27.664955 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e91765-38cf-4ca2-abcb-eb7d271b5621 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:27.665725 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3e91765-38cf-4ca2-abcb-eb7d271b5621 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:27.668431 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3e91765-38cf-4ca2-abcb-eb7d271b5621 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:27.702828 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 969/1941] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:53:27 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:53:27.706372 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f72d6d4-4345-4fdb-a9f8-e69a4c80af59 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:27.723677 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f72d6d4-4345-4fdb-a9f8-e69a4c80af59 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:27.739521 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-953326cb-2cad-4c8b-803f-cfcab811f6b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:27.740221 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-953326cb-2cad-4c8b-803f-cfcab811f6b0 admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:27.740505 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-953326cb-2cad-4c8b-803f-cfcab811f6b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:27.740783 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-953326cb-2cad-4c8b-803f-cfcab811f6b0 admin admin] PATH_INFO: `/v3/projects/b8f3600375c04434802f40cfcf8af5dc/users/2a5b385434674c73aa0187e110e7210d/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:27.798354 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-953326cb-2cad-4c8b-803f-cfcab811f6b0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b8f3600375c04434802f40cfcf8af5dc, user_id=2a5b385434674c73aa0187e110e7210d, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:27.814523 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-953326cb-2cad-4c8b-803f-cfcab811f6b0 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:27.863918 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 973/1942] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:53:27 2022] PUT /identity/v3/projects/b8f3600375c04434802f40cfcf8af5dc/users/2a5b385434674c73aa0187e110e7210d/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:53:27.865205 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3e91765-38cf-4ca2-abcb-eb7d271b5621 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:27.881578 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3e91765-38cf-4ca2-abcb-eb7d271b5621 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:28.018642 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b16c2aa-346c-4a73-a810-65782cac31ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:28.019798 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b16c2aa-346c-4a73-a810-65782cac31ed admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:28.020294 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b16c2aa-346c-4a73-a810-65782cac31ed admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:28.020937 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b16c2aa-346c-4a73-a810-65782cac31ed admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:28.021844 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b16c2aa-346c-4a73-a810-65782cac31ed admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:28.023874 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b16c2aa-346c-4a73-a810-65782cac31ed admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:28.036367 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 970/1943] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:53:27 2022] GET /identity/v3/roles => generated 1420 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:53:28.040713 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-953326cb-2cad-4c8b-803f-cfcab811f6b0 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:28.069269 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b16c2aa-346c-4a73-a810-65782cac31ed admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:28.071534 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-7b16c2aa-346c-4a73-a810-65782cac31ed admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:28.072442 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-953326cb-2cad-4c8b-803f-cfcab811f6b0 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:28.078378 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61037871-cf4d-447a-b7c2-8b55b215a820 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:28.079171 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61037871-cf4d-447a-b7c2-8b55b215a820 admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:28.079452 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61037871-cf4d-447a-b7c2-8b55b215a820 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:28.079726 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61037871-cf4d-447a-b7c2-8b55b215a820 admin admin] PATH_INFO: `/v3/projects/b8f3600375c04434802f40cfcf8af5dc/users/2a5b385434674c73aa0187e110e7210d/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:28.081957 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61037871-cf4d-447a-b7c2-8b55b215a820 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b8f3600375c04434802f40cfcf8af5dc, user_id=2a5b385434674c73aa0187e110e7210d, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:28.084670 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61037871-cf4d-447a-b7c2-8b55b215a820 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:28.097524 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 974/1944] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:53:28 2022] PUT /identity/v3/projects/b8f3600375c04434802f40cfcf8af5dc/users/2a5b385434674c73aa0187e110e7210d/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:28.108753 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69c15c8-20e7-4b80-a29e-8d95ecc5d87c None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:28.109038 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69c15c8-20e7-4b80-a29e-8d95ecc5d87c None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:28.109309 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69c15c8-20e7-4b80-a29e-8d95ecc5d87c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:28.142174 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-e69c15c8-20e7-4b80-a29e-8d95ecc5d87c None None] MFA Rules not processed for user `2a5b385434674c73aa0187e110e7210d`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:28.143602 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-e69c15c8-20e7-4b80-a29e-8d95ecc5d87c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:28.180327 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b16c2aa-346c-4a73-a810-65782cac31ed admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:28.239387 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d6dd7d2-f379-4e33-8937-74b2bc8cb7c0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:28.239930 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d6dd7d2-f379-4e33-8937-74b2bc8cb7c0 service cinder] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:28.240260 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d6dd7d2-f379-4e33-8937-74b2bc8cb7c0 service cinder] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:28.240598 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d6dd7d2-f379-4e33-8937-74b2bc8cb7c0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:28.242581 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-6d6dd7d2-f379-4e33-8937-74b2bc8cb7c0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:28.246542 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 975/1945] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:28 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:53:28.304484 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:28.319828 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d6dd7d2-f379-4e33-8937-74b2bc8cb7c0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:28.321613 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d6dd7d2-f379-4e33-8937-74b2bc8cb7c0 service cinder] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:28.336805 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e69c15c8-20e7-4b80-a29e-8d95ecc5d87c None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:28.341522 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-e69c15c8-20e7-4b80-a29e-8d95ecc5d87c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:28.349669 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 971/1946] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:53:28 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 281 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:53:28.354779 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e69c15c8-20e7-4b80-a29e-8d95ecc5d87c None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:28.359635 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85608167-88c9-4310-8861-e896fdb35cc5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:28.360506 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85608167-88c9-4310-8861-e896fdb35cc5 service cinder] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:28.360886 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85608167-88c9-4310-8861-e896fdb35cc5 service cinder] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:28.361254 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85608167-88c9-4310-8861-e896fdb35cc5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:28.373030 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85608167-88c9-4310-8861-e896fdb35cc5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:28.374846 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85608167-88c9-4310-8861-e896fdb35cc5 service cinder] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:28.390965 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 976/1947] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:53:28 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 55 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 11 00:53:28.486099 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d6dd7d2-f379-4e33-8937-74b2bc8cb7c0 service cinder] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:28.496753 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d6dd7d2-f379-4e33-8937-74b2bc8cb7c0 service cinder] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:28.521810 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32053999-8b03-49a7-8221-34e128fc1266 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:28.522651 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32053999-8b03-49a7-8221-34e128fc1266 service cinder] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:28.522957 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32053999-8b03-49a7-8221-34e128fc1266 service cinder] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:28.523300 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32053999-8b03-49a7-8221-34e128fc1266 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:28.638735 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32053999-8b03-49a7-8221-34e128fc1266 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:28.640533 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32053999-8b03-49a7-8221-34e128fc1266 service cinder] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:28.651098 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 972/1948] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:53:28 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 166 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:53:29.235711 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85608167-88c9-4310-8861-e896fdb35cc5 service cinder] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:29.237262 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-85608167-88c9-4310-8861-e896fdb35cc5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:29.291214 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:29.306268 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85608167-88c9-4310-8861-e896fdb35cc5 service cinder] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:29.311960 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1072593-2bcf-42a9-812d-ae44ae0a4415 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:29.312670 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1072593-2bcf-42a9-812d-ae44ae0a4415 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:29.313036 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1072593-2bcf-42a9-812d-ae44ae0a4415 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:29.313308 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1072593-2bcf-42a9-812d-ae44ae0a4415 admin admin] PATH_INFO: `/v3/users/6e566054861c4da7af7b4f8d3c5464c2` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:29.314424 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1072593-2bcf-42a9-812d-ae44ae0a4415 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6e566054861c4da7af7b4f8d3c5464c2)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:29.316383 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1072593-2bcf-42a9-812d-ae44ae0a4415 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:29.383381 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-c1072593-2bcf-42a9-812d-ae44ae0a4415 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6e566054861c4da7af7b4f8d3c5464c2'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:29.384111 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-c1072593-2bcf-42a9-812d-ae44ae0a4415 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6e566054861c4da7af7b4f8d3c5464c2'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:29.384920 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-c1072593-2bcf-42a9-812d-ae44ae0a4415 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6e566054861c4da7af7b4f8d3c5464c2'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:29.393744 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-c1072593-2bcf-42a9-812d-ae44ae0a4415 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6e566054861c4da7af7b4f8d3c5464c2'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:29.405773 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-c1072593-2bcf-42a9-812d-ae44ae0a4415 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6e566054861c4da7af7b4f8d3c5464c2'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:29.418618 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 977/1949] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:53:29 2022] DELETE /identity/v3/users/6e566054861c4da7af7b4f8d3c5464c2 => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:29.423374 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7645f094-83b4-458f-9f44-257d57619fae None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:29.423852 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7645f094-83b4-458f-9f44-257d57619fae None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:29.423984 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7645f094-83b4-458f-9f44-257d57619fae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:29.443253 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-7645f094-83b4-458f-9f44-257d57619fae None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:29.444444 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-7645f094-83b4-458f-9f44-257d57619fae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:29.502585 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 973/1950] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:53:29.512312 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1072593-2bcf-42a9-812d-ae44ae0a4415 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:29.514118 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-c1072593-2bcf-42a9-812d-ae44ae0a4415 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:29.574350 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1072593-2bcf-42a9-812d-ae44ae0a4415 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:29.578633 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1bc026-7e3e-468d-ada1-055dd4bcffe2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:29.579271 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1bc026-7e3e-468d-ada1-055dd4bcffe2 service neutron] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:29.579556 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1bc026-7e3e-468d-ada1-055dd4bcffe2 service neutron] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:29.579828 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1bc026-7e3e-468d-ada1-055dd4bcffe2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:29.588896 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e1bc026-7e3e-468d-ada1-055dd4bcffe2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:29.590343 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e1bc026-7e3e-468d-ada1-055dd4bcffe2 service neutron] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:29.600495 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 978/1951] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:53:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:53:29.924616 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7645f094-83b4-458f-9f44-257d57619fae None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:29.926174 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-7645f094-83b4-458f-9f44-257d57619fae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:29.940115 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7645f094-83b4-458f-9f44-257d57619fae None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:29.945101 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecde2caf-398c-4813-a246-6827e9d2d55c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:29.945866 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecde2caf-398c-4813-a246-6827e9d2d55c admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:29.946153 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecde2caf-398c-4813-a246-6827e9d2d55c admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:29.946550 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecde2caf-398c-4813-a246-6827e9d2d55c admin admin] PATH_INFO: `/v3/projects/f1929149e6454ceab2bbb044f38b711a` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:29.947926 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecde2caf-398c-4813-a246-6827e9d2d55c admin admin] RBAC: Authorizing `identity:delete_project(project_id=f1929149e6454ceab2bbb044f38b711a)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:29.949833 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecde2caf-398c-4813-a246-6827e9d2d55c admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:30.011652 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-ecde2caf-398c-4813-a246-6827e9d2d55c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f1929149e6454ceab2bbb044f38b711a', 'request_id': 'req-ecde2caf-398c-4813-a246-6827e9d2d55c'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:30.020281 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-ecde2caf-398c-4813-a246-6827e9d2d55c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f1929149e6454ceab2bbb044f38b711a', 'request_id': 'req-ecde2caf-398c-4813-a246-6827e9d2d55c'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:30.030502 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-ecde2caf-398c-4813-a246-6827e9d2d55c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f1929149e6454ceab2bbb044f38b711a', 'request_id': 'req-ecde2caf-398c-4813-a246-6827e9d2d55c'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:30.042460 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-ecde2caf-398c-4813-a246-6827e9d2d55c admin admin] The token cache is being invalidate because project f1929149e6454ceab2bbb044f38b711a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87924) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:53:30.042748 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-ecde2caf-398c-4813-a246-6827e9d2d55c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:30.043397 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-ecde2caf-398c-4813-a246-6827e9d2d55c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:30.045877 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 974/1952] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:53:29 2022] DELETE /identity/v3/projects/f1929149e6454ceab2bbb044f38b711a => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:30.049884 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1bc026-7e3e-468d-ada1-055dd4bcffe2 service neutron] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:30.059541 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1bc026-7e3e-468d-ada1-055dd4bcffe2 service neutron] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:30.151548 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4b9a71f-622c-48e2-ba48-2b55db57ad1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:30.151548 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b9a71f-622c-48e2-ba48-2b55db57ad1e admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:30.151548 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b9a71f-622c-48e2-ba48-2b55db57ad1e admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:30.151892 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b9a71f-622c-48e2-ba48-2b55db57ad1e admin admin] PATH_INFO: `/v3/users/3b341df142a04390938460ec5bd0718f` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:30.153892 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4b9a71f-622c-48e2-ba48-2b55db57ad1e admin admin] RBAC: Authorizing `identity:delete_user(user_id=3b341df142a04390938460ec5bd0718f)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:30.157211 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4b9a71f-622c-48e2-ba48-2b55db57ad1e admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:30.176982 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:30.235491 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-d4b9a71f-622c-48e2-ba48-2b55db57ad1e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3b341df142a04390938460ec5bd0718f'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:30.236045 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-d4b9a71f-622c-48e2-ba48-2b55db57ad1e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3b341df142a04390938460ec5bd0718f'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:30.237736 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-d4b9a71f-622c-48e2-ba48-2b55db57ad1e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3b341df142a04390938460ec5bd0718f'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:30.246181 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-d4b9a71f-622c-48e2-ba48-2b55db57ad1e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3b341df142a04390938460ec5bd0718f'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:30.265837 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-d4b9a71f-622c-48e2-ba48-2b55db57ad1e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3b341df142a04390938460ec5bd0718f'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:30.285877 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 979/1953] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:53:30 2022] DELETE /identity/v3/users/3b341df142a04390938460ec5bd0718f => generated 0 bytes in 237 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:31.253338 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecde2caf-398c-4813-a246-6827e9d2d55c admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:31.254848 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-ecde2caf-398c-4813-a246-6827e9d2d55c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:31.268808 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecde2caf-398c-4813-a246-6827e9d2d55c admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:31.320540 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:31.373718 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ee461bf-e7ad-4f52-b0b1-3e206e6f0263 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:31.374360 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee461bf-e7ad-4f52-b0b1-3e206e6f0263 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:31.374636 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee461bf-e7ad-4f52-b0b1-3e206e6f0263 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:31.374969 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee461bf-e7ad-4f52-b0b1-3e206e6f0263 admin admin] PATH_INFO: `/v3/projects/f7ad2cb683ef49d1bd4697a38369b280` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:31.376112 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ee461bf-e7ad-4f52-b0b1-3e206e6f0263 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f7ad2cb683ef49d1bd4697a38369b280)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:31.378003 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ee461bf-e7ad-4f52-b0b1-3e206e6f0263 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:31.427688 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-8ee461bf-e7ad-4f52-b0b1-3e206e6f0263 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f7ad2cb683ef49d1bd4697a38369b280', 'request_id': 'req-8ee461bf-e7ad-4f52-b0b1-3e206e6f0263'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:31.433878 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-8ee461bf-e7ad-4f52-b0b1-3e206e6f0263 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f7ad2cb683ef49d1bd4697a38369b280', 'request_id': 'req-8ee461bf-e7ad-4f52-b0b1-3e206e6f0263'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:31.441813 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-8ee461bf-e7ad-4f52-b0b1-3e206e6f0263 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f7ad2cb683ef49d1bd4697a38369b280', 'request_id': 'req-8ee461bf-e7ad-4f52-b0b1-3e206e6f0263'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:31.454069 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-8ee461bf-e7ad-4f52-b0b1-3e206e6f0263 admin admin] The token cache is being invalidate because project f7ad2cb683ef49d1bd4697a38369b280 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87924) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:53:31.454069 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-8ee461bf-e7ad-4f52-b0b1-3e206e6f0263 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:31.454525 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-8ee461bf-e7ad-4f52-b0b1-3e206e6f0263 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:31.456358 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 975/1954] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:53:31 2022] DELETE /identity/v3/projects/f7ad2cb683ef49d1bd4697a38369b280 => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:31.461427 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df0ea9e-228e-4771-b258-e809e32864a4 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:31.461869 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df0ea9e-228e-4771-b258-e809e32864a4 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:31.462209 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df0ea9e-228e-4771-b258-e809e32864a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:31.485275 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-2df0ea9e-228e-4771-b258-e809e32864a4 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:31.487187 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-2df0ea9e-228e-4771-b258-e809e32864a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:31.554620 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 980/1955] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:53:31.564140 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e26b5d-766c-4fe8-94a8-49f89128bf7a None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:31.564419 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e26b5d-766c-4fe8-94a8-49f89128bf7a None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:31.564691 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e26b5d-766c-4fe8-94a8-49f89128bf7a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:31.585847 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-48e26b5d-766c-4fe8-94a8-49f89128bf7a None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:31.586942 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-48e26b5d-766c-4fe8-94a8-49f89128bf7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:31.597159 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 976/1956] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:53:31.601465 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df0ea9e-228e-4771-b258-e809e32864a4 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:31.611398 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df0ea9e-228e-4771-b258-e809e32864a4 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:31.616122 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b3b431-f71f-4548-8e4e-53c9bf769e78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:31.616716 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b3b431-f71f-4548-8e4e-53c9bf769e78 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:31.616987 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b3b431-f71f-4548-8e4e-53c9bf769e78 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:31.617259 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b3b431-f71f-4548-8e4e-53c9bf769e78 admin admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:31.617898 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63b3b431-f71f-4548-8e4e-53c9bf769e78 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:53:31.618194 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63b3b431-f71f-4548-8e4e-53c9bf769e78 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:31.619822 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63b3b431-f71f-4548-8e4e-53c9bf769e78 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:31.631992 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 981/1957] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 00:53:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:53:31.636431 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48e26b5d-766c-4fe8-94a8-49f89128bf7a None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:31.653951 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48e26b5d-766c-4fe8-94a8-49f89128bf7a None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:31.663658 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e068377-86a1-4a8e-8d87-d5cafa60fd33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:31.664737 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e068377-86a1-4a8e-8d87-d5cafa60fd33 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:31.665082 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e068377-86a1-4a8e-8d87-d5cafa60fd33 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:31.665279 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e068377-86a1-4a8e-8d87-d5cafa60fd33 admin admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:31.666122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e068377-86a1-4a8e-8d87-d5cafa60fd33 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:31.669283 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e068377-86a1-4a8e-8d87-d5cafa60fd33 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:31.700777 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 977/1958] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:53:31 2022] POST /identity/v3/projects => generated 385 bytes in 65 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:53:31.706417 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b3b431-f71f-4548-8e4e-53c9bf769e78 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:31.730072 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b3b431-f71f-4548-8e4e-53c9bf769e78 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:31.807932 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc4e2044-5774-46d0-9898-dd20078772f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:31.808319 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4e2044-5774-46d0-9898-dd20078772f4 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:31.808570 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4e2044-5774-46d0-9898-dd20078772f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:31.808835 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4e2044-5774-46d0-9898-dd20078772f4 admin admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:31.809560 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc4e2044-5774-46d0-9898-dd20078772f4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:31.812095 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc4e2044-5774-46d0-9898-dd20078772f4 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:31.845280 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 982/1959] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:53:31 2022] POST /identity/v3/users => generated 293 bytes in 141 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:53:31.849546 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e068377-86a1-4a8e-8d87-d5cafa60fd33 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:31.866114 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e068377-86a1-4a8e-8d87-d5cafa60fd33 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:31.873192 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b3ea50b-8fbe-4afb-b7ec-2b9395f858ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:31.874283 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b3ea50b-8fbe-4afb-b7ec-2b9395f858ba admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:31.874633 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b3ea50b-8fbe-4afb-b7ec-2b9395f858ba admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:31.875014 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b3ea50b-8fbe-4afb-b7ec-2b9395f858ba admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:31.875776 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b3ea50b-8fbe-4afb-b7ec-2b9395f858ba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:31.877497 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b3ea50b-8fbe-4afb-b7ec-2b9395f858ba admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:31.891538 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 978/1960] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:53:31 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:53:31.895432 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc4e2044-5774-46d0-9898-dd20078772f4 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:31.906608 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc4e2044-5774-46d0-9898-dd20078772f4 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:31.912114 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5451a860-81bb-44e0-98ff-5c996058d59b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:31.912285 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5451a860-81bb-44e0-98ff-5c996058d59b admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:31.912502 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5451a860-81bb-44e0-98ff-5c996058d59b admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:31.912776 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5451a860-81bb-44e0-98ff-5c996058d59b admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:31.913421 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5451a860-81bb-44e0-98ff-5c996058d59b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:31.915219 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5451a860-81bb-44e0-98ff-5c996058d59b admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:31.918033 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=87,keystone:DELETE=9,keystone:INSERT=4 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:31.928496 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 983/1961] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:53:31 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:53:31.932437 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b3ea50b-8fbe-4afb-b7ec-2b9395f858ba admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:31.947011 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b3ea50b-8fbe-4afb-b7ec-2b9395f858ba admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:31.954429 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aef6e50b-2a9f-4659-a0a1-5f19e65a7df4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:31.955941 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aef6e50b-2a9f-4659-a0a1-5f19e65a7df4 admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:31.956467 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aef6e50b-2a9f-4659-a0a1-5f19e65a7df4 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:31.957344 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aef6e50b-2a9f-4659-a0a1-5f19e65a7df4 admin admin] PATH_INFO: `/v3/projects/9f6e6907d36b4ccabc38ab5b86e495c7/users/bb62b85efc944fb4ac501510923c05fe/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:31.980473 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aef6e50b-2a9f-4659-a0a1-5f19e65a7df4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9f6e6907d36b4ccabc38ab5b86e495c7, user_id=bb62b85efc944fb4ac501510923c05fe, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:31.984019 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aef6e50b-2a9f-4659-a0a1-5f19e65a7df4 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:32.000484 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 979/1962] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:53:31 2022] PUT /identity/v3/projects/9f6e6907d36b4ccabc38ab5b86e495c7/users/bb62b85efc944fb4ac501510923c05fe/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:32.010813 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3d29a8-190d-44c5-b126-0c6a0bddfcbb None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:32.011139 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3d29a8-190d-44c5-b126-0c6a0bddfcbb None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:32.011361 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3d29a8-190d-44c5-b126-0c6a0bddfcbb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:32.049198 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-fa3d29a8-190d-44c5-b126-0c6a0bddfcbb None None] MFA Rules not processed for user `bb62b85efc944fb4ac501510923c05fe`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:32.049198 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-fa3d29a8-190d-44c5-b126-0c6a0bddfcbb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:32.153487 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 984/1963] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:32 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:53:32.154370 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aef6e50b-2a9f-4659-a0a1-5f19e65a7df4 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:32.219283 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:32.246822 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aef6e50b-2a9f-4659-a0a1-5f19e65a7df4 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:32.253042 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f4839ed-1c8c-4d6f-956b-4bca56592534 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:32.253711 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f4839ed-1c8c-4d6f-956b-4bca56592534 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:32.254175 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f4839ed-1c8c-4d6f-956b-4bca56592534 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:32.254175 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f4839ed-1c8c-4d6f-956b-4bca56592534 admin admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:32.255242 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f4839ed-1c8c-4d6f-956b-4bca56592534 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:32.259362 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f4839ed-1c8c-4d6f-956b-4bca56592534 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:32.305152 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 980/1964] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:53:32 2022] POST /identity/v3/projects => generated 387 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:53:32.308955 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa3d29a8-190d-44c5-b126-0c6a0bddfcbb None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:32.330128 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa3d29a8-190d-44c5-b126-0c6a0bddfcbb None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:32.411512 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-608ee947-035e-400d-bd12-fa95843112a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:32.412689 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608ee947-035e-400d-bd12-fa95843112a7 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:32.413171 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608ee947-035e-400d-bd12-fa95843112a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:32.413572 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608ee947-035e-400d-bd12-fa95843112a7 admin admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:32.415700 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-608ee947-035e-400d-bd12-fa95843112a7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:32.417551 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-608ee947-035e-400d-bd12-fa95843112a7 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:32.473564 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 985/1965] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:53:32 2022] POST /identity/v3/users => generated 300 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:53:32.474806 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f4839ed-1c8c-4d6f-956b-4bca56592534 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:32.486548 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f4839ed-1c8c-4d6f-956b-4bca56592534 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:32.491795 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f0a6c9a-0b39-4afb-846e-7745ac561275 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:32.492414 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0a6c9a-0b39-4afb-846e-7745ac561275 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:32.492706 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0a6c9a-0b39-4afb-846e-7745ac561275 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:32.492964 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0a6c9a-0b39-4afb-846e-7745ac561275 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:32.493610 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f0a6c9a-0b39-4afb-846e-7745ac561275 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:32.495254 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f0a6c9a-0b39-4afb-846e-7745ac561275 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:32.515723 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 981/1966] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:53:32 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 11 00:53:32.517191 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-608ee947-035e-400d-bd12-fa95843112a7 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:32.541419 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-608ee947-035e-400d-bd12-fa95843112a7 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:32.551398 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6458f62-8e64-4dda-b39c-4349bd5dffb4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:32.552285 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6458f62-8e64-4dda-b39c-4349bd5dffb4 admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:32.552740 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6458f62-8e64-4dda-b39c-4349bd5dffb4 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:32.553158 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6458f62-8e64-4dda-b39c-4349bd5dffb4 admin admin] PATH_INFO: `/v3/projects/df2a5ece94924cc99127c28c492274d8/users/06b743f1252e4ba4830ef82c56659b20/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:32.578451 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6458f62-8e64-4dda-b39c-4349bd5dffb4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=df2a5ece94924cc99127c28c492274d8, user_id=06b743f1252e4ba4830ef82c56659b20, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:32.581450 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6458f62-8e64-4dda-b39c-4349bd5dffb4 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:32.600812 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 986/1967] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:53:32 2022] PUT /identity/v3/projects/df2a5ece94924cc99127c28c492274d8/users/06b743f1252e4ba4830ef82c56659b20/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 84 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:32.605033 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f0a6c9a-0b39-4afb-846e-7745ac561275 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:32.619174 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f0a6c9a-0b39-4afb-846e-7745ac561275 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:32.684012 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6458f62-8e64-4dda-b39c-4349bd5dffb4 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:32.685962 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-e6458f62-8e64-4dda-b39c-4349bd5dffb4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:32.735046 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9772d075-46e8-493b-9bad-e55c64064978 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:32.735670 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9772d075-46e8-493b-9bad-e55c64064978 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:32.735951 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9772d075-46e8-493b-9bad-e55c64064978 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:32.736218 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9772d075-46e8-493b-9bad-e55c64064978 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:32.736869 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9772d075-46e8-493b-9bad-e55c64064978 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:32.738581 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9772d075-46e8-493b-9bad-e55c64064978 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:32.779448 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 982/1968] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:53:32 2022] GET /identity/v3/roles => generated 1420 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 11 00:53:32.785505 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9772d075-46e8-493b-9bad-e55c64064978 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:32.805183 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:32.814443 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9772d075-46e8-493b-9bad-e55c64064978 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:32.822290 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ab797f-be1b-45f5-ad22-cc0e58ca9e1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:32.823141 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ab797f-be1b-45f5-ad22-cc0e58ca9e1d admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:32.823353 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ab797f-be1b-45f5-ad22-cc0e58ca9e1d admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:32.823627 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ab797f-be1b-45f5-ad22-cc0e58ca9e1d admin admin] PATH_INFO: `/v3/projects/df2a5ece94924cc99127c28c492274d8/users/06b743f1252e4ba4830ef82c56659b20/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:32.836827 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6458f62-8e64-4dda-b39c-4349bd5dffb4 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:32.841760 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ab797f-be1b-45f5-ad22-cc0e58ca9e1d admin admin] RBAC: Authorizing `identity:create_grant(project_id=df2a5ece94924cc99127c28c492274d8, user_id=06b743f1252e4ba4830ef82c56659b20, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:32.844695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ab797f-be1b-45f5-ad22-cc0e58ca9e1d admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:32.849679 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-952191da-1511-49a3-b169-7c1db7caef64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:32.851022 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952191da-1511-49a3-b169-7c1db7caef64 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:32.852524 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952191da-1511-49a3-b169-7c1db7caef64 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:32.852888 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952191da-1511-49a3-b169-7c1db7caef64 admin admin] PATH_INFO: `/v3/users/28d245fb98df47d889c0685e36cc5f22` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:32.856409 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 983/1969] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:53:32 2022] PUT /identity/v3/projects/df2a5ece94924cc99127c28c492274d8/users/06b743f1252e4ba4830ef82c56659b20/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 72 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:32.856690 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-952191da-1511-49a3-b169-7c1db7caef64 admin admin] RBAC: Authorizing `identity:delete_user(user_id=28d245fb98df47d889c0685e36cc5f22)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:32.858949 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-952191da-1511-49a3-b169-7c1db7caef64 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:32.866940 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c11074d-4b01-4f17-84b4-00dbb87a707f None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:32.867245 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c11074d-4b01-4f17-84b4-00dbb87a707f None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:32.867711 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c11074d-4b01-4f17-84b4-00dbb87a707f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:32.942047 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-6c11074d-4b01-4f17-84b4-00dbb87a707f None None] MFA Rules not processed for user `06b743f1252e4ba4830ef82c56659b20`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:32.945204 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-6c11074d-4b01-4f17-84b4-00dbb87a707f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:32.961915 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-952191da-1511-49a3-b169-7c1db7caef64 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '28d245fb98df47d889c0685e36cc5f22'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:32.962839 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-952191da-1511-49a3-b169-7c1db7caef64 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '28d245fb98df47d889c0685e36cc5f22'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:32.963678 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-952191da-1511-49a3-b169-7c1db7caef64 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '28d245fb98df47d889c0685e36cc5f22'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:32.972823 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-952191da-1511-49a3-b169-7c1db7caef64 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '28d245fb98df47d889c0685e36cc5f22'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:32.993869 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-952191da-1511-49a3-b169-7c1db7caef64 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '28d245fb98df47d889c0685e36cc5f22'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:33.021131 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 987/1970] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:53:32 2022] DELETE /identity/v3/users/28d245fb98df47d889c0685e36cc5f22 => generated 0 bytes in 338 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:33.027601 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b74227-6e86-4b84-bbce-6171b7d46804 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:33.027923 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b74227-6e86-4b84-bbce-6171b7d46804 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:33.028199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b74227-6e86-4b84-bbce-6171b7d46804 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:33.080626 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-72b74227-6e86-4b84-bbce-6171b7d46804 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:33.081153 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-72b74227-6e86-4b84-bbce-6171b7d46804 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:33.128867 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 984/1971] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:32 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 263 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:53:33.182639 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c11074d-4b01-4f17-84b4-00dbb87a707f None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:33.183306 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 988/1972] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:53:33.193352 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-6c11074d-4b01-4f17-84b4-00dbb87a707f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:33.194061 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b74227-6e86-4b84-bbce-6171b7d46804 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:33.195854 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-72b74227-6e86-4b84-bbce-6171b7d46804 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:33.226503 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:33.285985 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c11074d-4b01-4f17-84b4-00dbb87a707f None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:33.311968 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b74227-6e86-4b84-bbce-6171b7d46804 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:33.318099 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c4ff0b-f8ed-45de-bc9a-02700b794c52 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:33.318895 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c4ff0b-f8ed-45de-bc9a-02700b794c52 service neutron] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:33.319291 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c4ff0b-f8ed-45de-bc9a-02700b794c52 service neutron] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:33.319659 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c4ff0b-f8ed-45de-bc9a-02700b794c52 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:33.333580 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02c4ff0b-f8ed-45de-bc9a-02700b794c52 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:33.335770 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02c4ff0b-f8ed-45de-bc9a-02700b794c52 service neutron] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:33.351093 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 989/1973] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:53:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 158 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 11 00:53:33.363994 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-308cbaf1-4f2d-4d32-8d54-7bf48453a092 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:33.364731 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308cbaf1-4f2d-4d32-8d54-7bf48453a092 service cinder] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:33.365110 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308cbaf1-4f2d-4d32-8d54-7bf48453a092 service cinder] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:33.365490 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308cbaf1-4f2d-4d32-8d54-7bf48453a092 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:33.450932 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-308cbaf1-4f2d-4d32-8d54-7bf48453a092 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:33.452378 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-308cbaf1-4f2d-4d32-8d54-7bf48453a092 service cinder] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:33.464200 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 985/1974] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:53:33 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 282 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:53:33.775950 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c4ff0b-f8ed-45de-bc9a-02700b794c52 service neutron] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:33.779206 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-02c4ff0b-f8ed-45de-bc9a-02700b794c52 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:33.795508 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c4ff0b-f8ed-45de-bc9a-02700b794c52 service neutron] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:33.802402 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63763617-9ded-4186-a40d-36b1f3ef7aa0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:33.804917 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63763617-9ded-4186-a40d-36b1f3ef7aa0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:33.805210 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63763617-9ded-4186-a40d-36b1f3ef7aa0 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:33.805482 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63763617-9ded-4186-a40d-36b1f3ef7aa0 admin admin] PATH_INFO: `/v3/projects/733d0b2534874180a901f6b9eb352ff4` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:33.806699 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63763617-9ded-4186-a40d-36b1f3ef7aa0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=733d0b2534874180a901f6b9eb352ff4)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:33.809749 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63763617-9ded-4186-a40d-36b1f3ef7aa0 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:33.828244 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:33.873720 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-63763617-9ded-4186-a40d-36b1f3ef7aa0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '733d0b2534874180a901f6b9eb352ff4', 'request_id': 'req-63763617-9ded-4186-a40d-36b1f3ef7aa0'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:33.881147 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-63763617-9ded-4186-a40d-36b1f3ef7aa0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '733d0b2534874180a901f6b9eb352ff4', 'request_id': 'req-63763617-9ded-4186-a40d-36b1f3ef7aa0'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:33.887286 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-63763617-9ded-4186-a40d-36b1f3ef7aa0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '733d0b2534874180a901f6b9eb352ff4', 'request_id': 'req-63763617-9ded-4186-a40d-36b1f3ef7aa0'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:33.908811 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-63763617-9ded-4186-a40d-36b1f3ef7aa0 admin admin] The token cache is being invalidate because project 733d0b2534874180a901f6b9eb352ff4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87923) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:53:33.909294 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-63763617-9ded-4186-a40d-36b1f3ef7aa0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:33.910403 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-63763617-9ded-4186-a40d-36b1f3ef7aa0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:33.912389 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 990/1975] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:53:33 2022] DELETE /identity/v3/projects/733d0b2534874180a901f6b9eb352ff4 => generated 0 bytes in 138 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:53:33.916737 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-308cbaf1-4f2d-4d32-8d54-7bf48453a092 service cinder] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:33.939553 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-308cbaf1-4f2d-4d32-8d54-7bf48453a092 service cinder] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:34.015303 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a2c5bc-ebf3-4f32-b0b6-9e555f9ed1e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:34.015956 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a2c5bc-ebf3-4f32-b0b6-9e555f9ed1e2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:34.016689 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a2c5bc-ebf3-4f32-b0b6-9e555f9ed1e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:34.017008 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a2c5bc-ebf3-4f32-b0b6-9e555f9ed1e2 admin admin] PATH_INFO: `/v3/users/2a5b385434674c73aa0187e110e7210d` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:34.018292 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a2c5bc-ebf3-4f32-b0b6-9e555f9ed1e2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2a5b385434674c73aa0187e110e7210d)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:34.020726 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a2c5bc-ebf3-4f32-b0b6-9e555f9ed1e2 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:34.063420 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:34.098152 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-62a2c5bc-ebf3-4f32-b0b6-9e555f9ed1e2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2a5b385434674c73aa0187e110e7210d'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:34.106487 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-62a2c5bc-ebf3-4f32-b0b6-9e555f9ed1e2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2a5b385434674c73aa0187e110e7210d'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:34.107209 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-62a2c5bc-ebf3-4f32-b0b6-9e555f9ed1e2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2a5b385434674c73aa0187e110e7210d'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:34.115393 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-62a2c5bc-ebf3-4f32-b0b6-9e555f9ed1e2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2a5b385434674c73aa0187e110e7210d'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:34.129509 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-62a2c5bc-ebf3-4f32-b0b6-9e555f9ed1e2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2a5b385434674c73aa0187e110e7210d'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:34.149151 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 986/1976] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:53:33 2022] DELETE /identity/v3/users/2a5b385434674c73aa0187e110e7210d => generated 0 bytes in 233 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:34.197691 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63763617-9ded-4186-a40d-36b1f3ef7aa0 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:34.199148 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-63763617-9ded-4186-a40d-36b1f3ef7aa0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:34.213209 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63763617-9ded-4186-a40d-36b1f3ef7aa0 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:34.282705 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a0f1ee3-6a25-418d-8520-b486bba5b585 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:34.283388 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0f1ee3-6a25-418d-8520-b486bba5b585 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:34.283669 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0f1ee3-6a25-418d-8520-b486bba5b585 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:34.283946 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0f1ee3-6a25-418d-8520-b486bba5b585 admin admin] PATH_INFO: `/v3/users/bb62b85efc944fb4ac501510923c05fe` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:34.285351 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a0f1ee3-6a25-418d-8520-b486bba5b585 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bb62b85efc944fb4ac501510923c05fe)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:34.287346 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a0f1ee3-6a25-418d-8520-b486bba5b585 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:34.338129 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-0a0f1ee3-6a25-418d-8520-b486bba5b585 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bb62b85efc944fb4ac501510923c05fe'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:34.338830 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-0a0f1ee3-6a25-418d-8520-b486bba5b585 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bb62b85efc944fb4ac501510923c05fe'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:34.339618 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-0a0f1ee3-6a25-418d-8520-b486bba5b585 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bb62b85efc944fb4ac501510923c05fe'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:34.354125 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-0a0f1ee3-6a25-418d-8520-b486bba5b585 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bb62b85efc944fb4ac501510923c05fe'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:34.372016 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-0a0f1ee3-6a25-418d-8520-b486bba5b585 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bb62b85efc944fb4ac501510923c05fe'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:34.398266 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 991/1977] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:53:34 2022] DELETE /identity/v3/users/bb62b85efc944fb4ac501510923c05fe => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:34.406173 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4092cf-b7db-474b-b620-e956bd598082 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:34.406830 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4092cf-b7db-474b-b620-e956bd598082 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:34.407028 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4092cf-b7db-474b-b620-e956bd598082 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:34.431097 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-3d4092cf-b7db-474b-b620-e956bd598082 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:34.432474 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-3d4092cf-b7db-474b-b620-e956bd598082 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:34.466447 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a0f1ee3-6a25-418d-8520-b486bba5b585 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:34.467866 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-0a0f1ee3-6a25-418d-8520-b486bba5b585 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:34.494481 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:34.519385 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 987/1978] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:53:34.530187 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d4092cf-b7db-474b-b620-e956bd598082 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:34.531776 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-3d4092cf-b7db-474b-b620-e956bd598082 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:34.552075 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a0f1ee3-6a25-418d-8520-b486bba5b585 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:34.557347 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3bce6ad-e032-4992-b430-42e9ffd150a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:34.557975 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3bce6ad-e032-4992-b430-42e9ffd150a5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:34.558248 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3bce6ad-e032-4992-b430-42e9ffd150a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:34.558515 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3bce6ad-e032-4992-b430-42e9ffd150a5 admin admin] PATH_INFO: `/v3/projects/b8f3600375c04434802f40cfcf8af5dc` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:34.560688 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3bce6ad-e032-4992-b430-42e9ffd150a5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b8f3600375c04434802f40cfcf8af5dc)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:34.562592 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3bce6ad-e032-4992-b430-42e9ffd150a5 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:34.598295 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d4092cf-b7db-474b-b620-e956bd598082 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:34.602278 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1abcd7b9-f849-4b15-832f-8077f9782e65 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:34.603024 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1abcd7b9-f849-4b15-832f-8077f9782e65 service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:34.603361 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1abcd7b9-f849-4b15-832f-8077f9782e65 service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:34.603689 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1abcd7b9-f849-4b15-832f-8077f9782e65 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:34.614492 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1abcd7b9-f849-4b15-832f-8077f9782e65 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:34.616058 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-a3bce6ad-e032-4992-b430-42e9ffd150a5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b8f3600375c04434802f40cfcf8af5dc', 'request_id': 'req-a3bce6ad-e032-4992-b430-42e9ffd150a5'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:34.616402 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1abcd7b9-f849-4b15-832f-8077f9782e65 service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:34.623508 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-a3bce6ad-e032-4992-b430-42e9ffd150a5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b8f3600375c04434802f40cfcf8af5dc', 'request_id': 'req-a3bce6ad-e032-4992-b430-42e9ffd150a5'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:34.630296 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-a3bce6ad-e032-4992-b430-42e9ffd150a5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b8f3600375c04434802f40cfcf8af5dc', 'request_id': 'req-a3bce6ad-e032-4992-b430-42e9ffd150a5'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:34.641605 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 988/1979] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:53:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:53:34.647798 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-a3bce6ad-e032-4992-b430-42e9ffd150a5 admin admin] The token cache is being invalidate because project b8f3600375c04434802f40cfcf8af5dc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87923) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:53:34.648269 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-a3bce6ad-e032-4992-b430-42e9ffd150a5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:34.649240 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-a3bce6ad-e032-4992-b430-42e9ffd150a5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:34.650874 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 992/1980] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:53:34 2022] DELETE /identity/v3/projects/b8f3600375c04434802f40cfcf8af5dc => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:53:34.655928 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d68f88e-9e29-49b8-af53-e8e1add893f5 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:34.656221 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d68f88e-9e29-49b8-af53-e8e1add893f5 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:34.656496 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d68f88e-9e29-49b8-af53-e8e1add893f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:34.690024 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-1d68f88e-9e29-49b8-af53-e8e1add893f5 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:34.691616 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-1d68f88e-9e29-49b8-af53-e8e1add893f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:34.795001 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:34.813301 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 989/1981] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:53:34.820957 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc2cf3d-174f-4461-a507-36ecfc0a34a9 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:34.821488 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc2cf3d-174f-4461-a507-36ecfc0a34a9 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:34.821488 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc2cf3d-174f-4461-a507-36ecfc0a34a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:34.860543 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-1fc2cf3d-174f-4461-a507-36ecfc0a34a9 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:34.861836 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-1fc2cf3d-174f-4461-a507-36ecfc0a34a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:34.871365 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 993/1982] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:53:34.876373 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d68f88e-9e29-49b8-af53-e8e1add893f5 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:34.886998 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d68f88e-9e29-49b8-af53-e8e1add893f5 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:34.892136 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21946afd-6854-4284-b814-8690b0a59568 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:34.892747 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21946afd-6854-4284-b814-8690b0a59568 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:34.893024 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21946afd-6854-4284-b814-8690b0a59568 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:34.893315 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21946afd-6854-4284-b814-8690b0a59568 admin admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:34.893941 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21946afd-6854-4284-b814-8690b0a59568 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:53:34.894242 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21946afd-6854-4284-b814-8690b0a59568 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:34.895977 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21946afd-6854-4284-b814-8690b0a59568 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:34.908940 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 990/1983] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 00:53:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:53:34.913277 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc2cf3d-174f-4461-a507-36ecfc0a34a9 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:34.938664 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc2cf3d-174f-4461-a507-36ecfc0a34a9 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:34.947254 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beb43e2b-f1ac-46b0-8179-55e7fbd32524 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:34.948120 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb43e2b-f1ac-46b0-8179-55e7fbd32524 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:34.948439 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb43e2b-f1ac-46b0-8179-55e7fbd32524 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:34.948723 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb43e2b-f1ac-46b0-8179-55e7fbd32524 admin admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:34.949357 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beb43e2b-f1ac-46b0-8179-55e7fbd32524 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:34.952384 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beb43e2b-f1ac-46b0-8179-55e7fbd32524 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:34.982610 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 994/1984] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:53:34 2022] POST /identity/v3/projects => generated 387 bytes in 70 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:53:34.987944 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21946afd-6854-4284-b814-8690b0a59568 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:34.996466 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beb43e2b-f1ac-46b0-8179-55e7fbd32524 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:34.998501 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-beb43e2b-f1ac-46b0-8179-55e7fbd32524 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:34.999805 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21946afd-6854-4284-b814-8690b0a59568 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:35.064458 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a28564e2-b156-4f0b-8bc6-8c261b1aa60b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:35.065070 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28564e2-b156-4f0b-8bc6-8c261b1aa60b admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:35.065349 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28564e2-b156-4f0b-8bc6-8c261b1aa60b admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:35.065620 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28564e2-b156-4f0b-8bc6-8c261b1aa60b admin admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:35.066264 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a28564e2-b156-4f0b-8bc6-8c261b1aa60b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:35.068201 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a28564e2-b156-4f0b-8bc6-8c261b1aa60b admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:35.070594 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beb43e2b-f1ac-46b0-8179-55e7fbd32524 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:35.075804 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8095f5f0-54bb-4b58-bad4-4a1f44eb5936 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:35.076425 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8095f5f0-54bb-4b58-bad4-4a1f44eb5936 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:35.076708 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8095f5f0-54bb-4b58-bad4-4a1f44eb5936 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:35.077265 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8095f5f0-54bb-4b58-bad4-4a1f44eb5936 admin admin] PATH_INFO: `/v3/projects/9f6e6907d36b4ccabc38ab5b86e495c7` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:35.078971 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8095f5f0-54bb-4b58-bad4-4a1f44eb5936 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9f6e6907d36b4ccabc38ab5b86e495c7)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:35.081991 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8095f5f0-54bb-4b58-bad4-4a1f44eb5936 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:35.102973 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 991/1985] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:53:34 2022] POST /identity/v3/users => generated 294 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:53:35.103936 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a28564e2-b156-4f0b-8bc6-8c261b1aa60b admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:35.110393 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:35.114609 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a28564e2-b156-4f0b-8bc6-8c261b1aa60b admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:35.119913 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e68df64-2f75-4454-bc96-6a26e1a94e5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:35.120830 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e68df64-2f75-4454-bc96-6a26e1a94e5c admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:35.121275 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e68df64-2f75-4454-bc96-6a26e1a94e5c admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:35.121591 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e68df64-2f75-4454-bc96-6a26e1a94e5c admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:35.122460 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e68df64-2f75-4454-bc96-6a26e1a94e5c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:35.124103 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e68df64-2f75-4454-bc96-6a26e1a94e5c admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:35.135081 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 992/1986] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:53:35 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 11 00:53:35.136782 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-8095f5f0-54bb-4b58-bad4-4a1f44eb5936 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9f6e6907d36b4ccabc38ab5b86e495c7', 'request_id': 'req-8095f5f0-54bb-4b58-bad4-4a1f44eb5936'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:35.140314 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e68df64-2f75-4454-bc96-6a26e1a94e5c admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:35.144214 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-8095f5f0-54bb-4b58-bad4-4a1f44eb5936 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9f6e6907d36b4ccabc38ab5b86e495c7', 'request_id': 'req-8095f5f0-54bb-4b58-bad4-4a1f44eb5936'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:35.150010 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-8095f5f0-54bb-4b58-bad4-4a1f44eb5936 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9f6e6907d36b4ccabc38ab5b86e495c7', 'request_id': 'req-8095f5f0-54bb-4b58-bad4-4a1f44eb5936'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:35.150665 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e68df64-2f75-4454-bc96-6a26e1a94e5c admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:35.163238 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-8095f5f0-54bb-4b58-bad4-4a1f44eb5936 admin admin] The token cache is being invalidate because project 9f6e6907d36b4ccabc38ab5b86e495c7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87923) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:53:35.163537 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-8095f5f0-54bb-4b58-bad4-4a1f44eb5936 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:35.164172 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-8095f5f0-54bb-4b58-bad4-4a1f44eb5936 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:35.165673 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 995/1987] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:53:34 2022] DELETE /identity/v3/projects/9f6e6907d36b4ccabc38ab5b86e495c7 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:35.169873 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8095f5f0-54bb-4b58-bad4-4a1f44eb5936 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:35.171228 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-8095f5f0-54bb-4b58-bad4-4a1f44eb5936 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:35.209477 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f566865-04ed-49a0-b49b-eb199248b0d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:35.210168 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f566865-04ed-49a0-b49b-eb199248b0d4 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:35.210553 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f566865-04ed-49a0-b49b-eb199248b0d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:35.210853 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f566865-04ed-49a0-b49b-eb199248b0d4 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:35.211491 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f566865-04ed-49a0-b49b-eb199248b0d4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:35.213126 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f566865-04ed-49a0-b49b-eb199248b0d4 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:35.223668 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 993/1988] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:53:35 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 11 00:53:35.227673 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f566865-04ed-49a0-b49b-eb199248b0d4 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:35.229341 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-9f566865-04ed-49a0-b49b-eb199248b0d4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:35.240931 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8095f5f0-54bb-4b58-bad4-4a1f44eb5936 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:35.241458 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f566865-04ed-49a0-b49b-eb199248b0d4 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:35.245704 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-928836c4-2a1d-4fd3-81fe-e4fc1fe50591 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:35.246003 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb6633b-4981-4a23-aa3b-1350ec82a53a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:35.246331 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928836c4-2a1d-4fd3-81fe-e4fc1fe50591 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:35.247175 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb6633b-4981-4a23-aa3b-1350ec82a53a admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:35.247305 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928836c4-2a1d-4fd3-81fe-e4fc1fe50591 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:35.247454 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb6633b-4981-4a23-aa3b-1350ec82a53a admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:35.247664 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928836c4-2a1d-4fd3-81fe-e4fc1fe50591 admin admin] PATH_INFO: `/v3/users/06b743f1252e4ba4830ef82c56659b20` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:35.247755 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb6633b-4981-4a23-aa3b-1350ec82a53a admin admin] PATH_INFO: `/v3/projects/86b786e1eacd4aceb8e86cee504756e9/users/a2e772a358e64578850b11fdcf09a403/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:35.248813 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-928836c4-2a1d-4fd3-81fe-e4fc1fe50591 admin admin] RBAC: Authorizing `identity:delete_user(user_id=06b743f1252e4ba4830ef82c56659b20)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:35.250736 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-928836c4-2a1d-4fd3-81fe-e4fc1fe50591 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:35.263654 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecb6633b-4981-4a23-aa3b-1350ec82a53a admin admin] RBAC: Authorizing `identity:create_grant(project_id=86b786e1eacd4aceb8e86cee504756e9, user_id=a2e772a358e64578850b11fdcf09a403, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:35.267133 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecb6633b-4981-4a23-aa3b-1350ec82a53a admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:35.282501 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 994/1989] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:53:35 2022] PUT /identity/v3/projects/86b786e1eacd4aceb8e86cee504756e9/users/a2e772a358e64578850b11fdcf09a403/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:53:35.303311 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f2545a-8ed0-4a2c-b8f8-8814db0bd0fa None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:35.303653 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f2545a-8ed0-4a2c-b8f8-8814db0bd0fa None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:35.303938 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f2545a-8ed0-4a2c-b8f8-8814db0bd0fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:35.318041 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-928836c4-2a1d-4fd3-81fe-e4fc1fe50591 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '06b743f1252e4ba4830ef82c56659b20'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:35.319395 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-928836c4-2a1d-4fd3-81fe-e4fc1fe50591 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '06b743f1252e4ba4830ef82c56659b20'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:35.320557 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-928836c4-2a1d-4fd3-81fe-e4fc1fe50591 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '06b743f1252e4ba4830ef82c56659b20'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:35.339065 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-928836c4-2a1d-4fd3-81fe-e4fc1fe50591 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '06b743f1252e4ba4830ef82c56659b20'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:35.342428 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-23f2545a-8ed0-4a2c-b8f8-8814db0bd0fa None None] MFA Rules not processed for user `a2e772a358e64578850b11fdcf09a403`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:35.344079 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-23f2545a-8ed0-4a2c-b8f8-8814db0bd0fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:35.354461 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-928836c4-2a1d-4fd3-81fe-e4fc1fe50591 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '06b743f1252e4ba4830ef82c56659b20'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:35.366287 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:35.371061 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 996/1990] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:53:35 2022] DELETE /identity/v3/users/06b743f1252e4ba4830ef82c56659b20 => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:53:35.422635 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 995/1991] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:35 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:53:35.427203 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-928836c4-2a1d-4fd3-81fe-e4fc1fe50591 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:35.428629 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-928836c4-2a1d-4fd3-81fe-e4fc1fe50591 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:35.468942 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:DELETE=17,keystone:SELECT=81,keystone:INSERT=2 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:35.508288 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-928836c4-2a1d-4fd3-81fe-e4fc1fe50591 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:35.513324 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-facdc673-ba20-41ec-a05d-d93b592d1386 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:35.513988 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-facdc673-ba20-41ec-a05d-d93b592d1386 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:35.514210 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-facdc673-ba20-41ec-a05d-d93b592d1386 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:35.514540 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-facdc673-ba20-41ec-a05d-d93b592d1386 admin admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:35.515247 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-facdc673-ba20-41ec-a05d-d93b592d1386 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:35.517154 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-facdc673-ba20-41ec-a05d-d93b592d1386 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:35.549823 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 997/1992] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:53:35 2022] POST /identity/v3/projects => generated 387 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:53:35.553564 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23f2545a-8ed0-4a2c-b8f8-8814db0bd0fa None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:35.619217 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23f2545a-8ed0-4a2c-b8f8-8814db0bd0fa None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:35.663700 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-facdc673-ba20-41ec-a05d-d93b592d1386 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:35.665686 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-facdc673-ba20-41ec-a05d-d93b592d1386 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:35.687483 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3abae0e-8ea2-4708-a797-39f387630b3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:35.688336 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3abae0e-8ea2-4708-a797-39f387630b3b admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:35.688805 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3abae0e-8ea2-4708-a797-39f387630b3b admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:35.689304 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3abae0e-8ea2-4708-a797-39f387630b3b admin admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:35.690115 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3abae0e-8ea2-4708-a797-39f387630b3b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:35.692947 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3abae0e-8ea2-4708-a797-39f387630b3b admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:35.716820 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 996/1993] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:53:35 2022] POST /identity/v3/users => generated 300 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:53:35.721487 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3abae0e-8ea2-4708-a797-39f387630b3b admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:35.733771 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3abae0e-8ea2-4708-a797-39f387630b3b admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:35.741080 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e2924e-5227-486a-bcb0-0dd1a0a41d09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:35.741704 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e2924e-5227-486a-bcb0-0dd1a0a41d09 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:35.741950 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-facdc673-ba20-41ec-a05d-d93b592d1386 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:35.742070 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e2924e-5227-486a-bcb0-0dd1a0a41d09 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:35.742303 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e2924e-5227-486a-bcb0-0dd1a0a41d09 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:35.743144 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96e2924e-5227-486a-bcb0-0dd1a0a41d09 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:35.745059 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96e2924e-5227-486a-bcb0-0dd1a0a41d09 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:35.747358 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87bd3910-7b61-4670-b107-9a865a8d453c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:35.748163 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87bd3910-7b61-4670-b107-9a865a8d453c admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:35.748504 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87bd3910-7b61-4670-b107-9a865a8d453c admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:35.749062 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87bd3910-7b61-4670-b107-9a865a8d453c admin admin] PATH_INFO: `/v3/projects/df2a5ece94924cc99127c28c492274d8` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:35.750314 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87bd3910-7b61-4670-b107-9a865a8d453c admin admin] RBAC: Authorizing `identity:delete_project(project_id=df2a5ece94924cc99127c28c492274d8)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:35.752318 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87bd3910-7b61-4670-b107-9a865a8d453c admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:35.765604 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 997/1994] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:53:35 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 11 00:53:35.766575 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e2924e-5227-486a-bcb0-0dd1a0a41d09 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:35.782732 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e2924e-5227-486a-bcb0-0dd1a0a41d09 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:35.789611 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8520b935-0aa3-46f9-8968-5b5f54955f70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:35.790444 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8520b935-0aa3-46f9-8968-5b5f54955f70 admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:35.791109 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8520b935-0aa3-46f9-8968-5b5f54955f70 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:35.791533 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8520b935-0aa3-46f9-8968-5b5f54955f70 admin admin] PATH_INFO: `/v3/projects/25b5c52f70104623ac36dbcd0c20bab5/users/38508871925d465cb24e32a83e98fe2b/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:35.809327 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-87bd3910-7b61-4670-b107-9a865a8d453c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'df2a5ece94924cc99127c28c492274d8', 'request_id': 'req-87bd3910-7b61-4670-b107-9a865a8d453c'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:35.811820 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8520b935-0aa3-46f9-8968-5b5f54955f70 admin admin] RBAC: Authorizing `identity:create_grant(project_id=25b5c52f70104623ac36dbcd0c20bab5, user_id=38508871925d465cb24e32a83e98fe2b, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:35.814556 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8520b935-0aa3-46f9-8968-5b5f54955f70 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:35.825440 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-87bd3910-7b61-4670-b107-9a865a8d453c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'df2a5ece94924cc99127c28c492274d8', 'request_id': 'req-87bd3910-7b61-4670-b107-9a865a8d453c'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:35.830809 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 998/1995] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:53:35 2022] PUT /identity/v3/projects/25b5c52f70104623ac36dbcd0c20bab5/users/38508871925d465cb24e32a83e98fe2b/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:53:35.832098 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-87bd3910-7b61-4670-b107-9a865a8d453c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'df2a5ece94924cc99127c28c492274d8', 'request_id': 'req-87bd3910-7b61-4670-b107-9a865a8d453c'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:35.834327 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8520b935-0aa3-46f9-8968-5b5f54955f70 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:35.846013 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-87bd3910-7b61-4670-b107-9a865a8d453c admin admin] The token cache is being invalidate because project df2a5ece94924cc99127c28c492274d8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87923) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:53:35.846366 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-87bd3910-7b61-4670-b107-9a865a8d453c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:35.846983 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-87bd3910-7b61-4670-b107-9a865a8d453c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:35.848498 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 998/1996] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:53:35 2022] DELETE /identity/v3/projects/df2a5ece94924cc99127c28c492274d8 => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:35.853186 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2304fdd4-265e-4492-8bd3-bc39a138b8e3 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:35.853472 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2304fdd4-265e-4492-8bd3-bc39a138b8e3 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:35.853748 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2304fdd4-265e-4492-8bd3-bc39a138b8e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:35.857287 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:35.874147 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-2304fdd4-265e-4492-8bd3-bc39a138b8e3 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:35.875528 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-2304fdd4-265e-4492-8bd3-bc39a138b8e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:35.912415 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8520b935-0aa3-46f9-8968-5b5f54955f70 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:35.917086 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c59809-f16e-45fb-baa3-35003306ba02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:35.917691 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c59809-f16e-45fb-baa3-35003306ba02 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:35.917967 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c59809-f16e-45fb-baa3-35003306ba02 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:35.918235 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c59809-f16e-45fb-baa3-35003306ba02 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:35.919009 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55c59809-f16e-45fb-baa3-35003306ba02 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:35.920827 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55c59809-f16e-45fb-baa3-35003306ba02 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:35.923859 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:35.932760 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 999/1997] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:53:35 2022] GET /identity/v3/roles => generated 1420 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 11 00:53:35.937307 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c59809-f16e-45fb-baa3-35003306ba02 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:35.941101 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-55c59809-f16e-45fb-baa3-35003306ba02 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:35.951787 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 999/1998] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:53:35.961026 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1716ec-affa-4f94-a6fd-2cf6193270a8 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:35.961026 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1716ec-affa-4f94-a6fd-2cf6193270a8 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:35.961026 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1716ec-affa-4f94-a6fd-2cf6193270a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:35.963655 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c59809-f16e-45fb-baa3-35003306ba02 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:35.970455 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efaeeb56-cbdc-44b1-a3cf-42c9964bfebc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:35.972181 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efaeeb56-cbdc-44b1-a3cf-42c9964bfebc admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:35.972527 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efaeeb56-cbdc-44b1-a3cf-42c9964bfebc admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:35.972807 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efaeeb56-cbdc-44b1-a3cf-42c9964bfebc admin admin] PATH_INFO: `/v3/projects/25b5c52f70104623ac36dbcd0c20bab5/users/38508871925d465cb24e32a83e98fe2b/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:35.974842 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efaeeb56-cbdc-44b1-a3cf-42c9964bfebc admin admin] RBAC: Authorizing `identity:create_grant(project_id=25b5c52f70104623ac36dbcd0c20bab5, user_id=38508871925d465cb24e32a83e98fe2b, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:35.977521 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efaeeb56-cbdc-44b1-a3cf-42c9964bfebc admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:35.992237 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-7b1716ec-affa-4f94-a6fd-2cf6193270a8 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:35.992579 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1000/1999] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:53:35 2022] PUT /identity/v3/projects/25b5c52f70104623ac36dbcd0c20bab5/users/38508871925d465cb24e32a83e98fe2b/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:35.993872 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-7b1716ec-affa-4f94-a6fd-2cf6193270a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:36.003589 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7fe08a1-27af-4910-8af8-12b3154e3258 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:36.003873 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7fe08a1-27af-4910-8af8-12b3154e3258 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:36.004145 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7fe08a1-27af-4910-8af8-12b3154e3258 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:36.035851 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-c7fe08a1-27af-4910-8af8-12b3154e3258 None None] MFA Rules not processed for user `38508871925d465cb24e32a83e98fe2b`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:36.037079 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-c7fe08a1-27af-4910-8af8-12b3154e3258 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:36.082298 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1000/2000] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:53:36.082298 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b1716ec-affa-4f94-a6fd-2cf6193270a8 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:36.094109 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b1716ec-affa-4f94-a6fd-2cf6193270a8 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:36.100733 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c70e17-393a-49ba-be69-1b6db8e3fe33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:36.101464 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c70e17-393a-49ba-be69-1b6db8e3fe33 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:36.101896 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c70e17-393a-49ba-be69-1b6db8e3fe33 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:36.102172 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c70e17-393a-49ba-be69-1b6db8e3fe33 admin admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:36.102902 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44c70e17-393a-49ba-be69-1b6db8e3fe33 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:53:36.103159 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44c70e17-393a-49ba-be69-1b6db8e3fe33 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:36.105012 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44c70e17-393a-49ba-be69-1b6db8e3fe33 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:36.120966 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1001/2001] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 00:53:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:53:36.124632 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1001/2002] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:36 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:53:36.125696 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c70e17-393a-49ba-be69-1b6db8e3fe33 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:36.135514 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7fe08a1-27af-4910-8af8-12b3154e3258 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:36.135907 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c70e17-393a-49ba-be69-1b6db8e3fe33 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:36.137073 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-c7fe08a1-27af-4910-8af8-12b3154e3258 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:36.140594 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc2de442-c0f0-4373-b201-4ef23c005e84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:36.141188 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2de442-c0f0-4373-b201-4ef23c005e84 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:36.141467 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2de442-c0f0-4373-b201-4ef23c005e84 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:36.141739 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2de442-c0f0-4373-b201-4ef23c005e84 admin admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:36.142427 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc2de442-c0f0-4373-b201-4ef23c005e84 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:36.144355 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc2de442-c0f0-4373-b201-4ef23c005e84 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:36.173421 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1002/2003] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:53:36 2022] POST /identity/v3/projects => generated 381 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:53:36.177556 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc2de442-c0f0-4373-b201-4ef23c005e84 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:36.202071 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7fe08a1-27af-4910-8af8-12b3154e3258 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:36.240817 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc2de442-c0f0-4373-b201-4ef23c005e84 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:36.245609 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4352be8-ddba-4321-bd4e-c9249ee43626 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:36.246120 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:36.246275 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4352be8-ddba-4321-bd4e-c9249ee43626 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:36.246709 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4352be8-ddba-4321-bd4e-c9249ee43626 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:36.246860 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4352be8-ddba-4321-bd4e-c9249ee43626 admin admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:36.247530 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4352be8-ddba-4321-bd4e-c9249ee43626 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:36.249588 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4352be8-ddba-4321-bd4e-c9249ee43626 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:36.255268 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e90d1c-73b6-4215-955b-f50d7e682ed7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:36.255905 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e90d1c-73b6-4215-955b-f50d7e682ed7 service cinder] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:36.256178 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e90d1c-73b6-4215-955b-f50d7e682ed7 service cinder] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:36.256446 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e90d1c-73b6-4215-955b-f50d7e682ed7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:36.271483 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1003/2004] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:53:36 2022] POST /identity/v3/users => generated 291 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:53:36.276074 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4352be8-ddba-4321-bd4e-c9249ee43626 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:36.286517 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4352be8-ddba-4321-bd4e-c9249ee43626 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:36.291141 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4f1e7b5-28a5-4a6b-97c4-bcedf4fb3ecb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:36.291718 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f1e7b5-28a5-4a6b-97c4-bcedf4fb3ecb admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:36.291996 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f1e7b5-28a5-4a6b-97c4-bcedf4fb3ecb admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:36.292263 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f1e7b5-28a5-4a6b-97c4-bcedf4fb3ecb admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:36.292895 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4f1e7b5-28a5-4a6b-97c4-bcedf4fb3ecb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:36.295176 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4f1e7b5-28a5-4a6b-97c4-bcedf4fb3ecb admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:36.305247 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1004/2005] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:53:36 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 11 00:53:36.309818 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4f1e7b5-28a5-4a6b-97c4-bcedf4fb3ecb admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:36.318733 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57e90d1c-73b6-4215-955b-f50d7e682ed7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:36.319290 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4f1e7b5-28a5-4a6b-97c4-bcedf4fb3ecb admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:36.320205 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57e90d1c-73b6-4215-955b-f50d7e682ed7 service cinder] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:36.323969 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-142ee634-6cf4-49f8-92be-d66854c43391 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:36.324549 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142ee634-6cf4-49f8-92be-d66854c43391 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:36.325556 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142ee634-6cf4-49f8-92be-d66854c43391 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:36.325833 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142ee634-6cf4-49f8-92be-d66854c43391 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:36.326468 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-142ee634-6cf4-49f8-92be-d66854c43391 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:36.328097 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-142ee634-6cf4-49f8-92be-d66854c43391 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:36.330905 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1002/2006] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:53:36 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 196 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 11 00:53:36.338744 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1005/2007] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:53:36 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 11 00:53:36.343152 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e90d1c-73b6-4215-955b-f50d7e682ed7 service cinder] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:36.353359 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e90d1c-73b6-4215-955b-f50d7e682ed7 service cinder] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:36.358702 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27de6596-a648-4100-8693-2a598d22486b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:36.359420 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27de6596-a648-4100-8693-2a598d22486b admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:36.359704 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27de6596-a648-4100-8693-2a598d22486b admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:36.359982 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27de6596-a648-4100-8693-2a598d22486b admin admin] PATH_INFO: `/v3/projects/0641dd2f31fb4b468e273de5ba93b2bd/users/dcf81e7c7942480caf4c6d9d27a5b147/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:36.360401 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-142ee634-6cf4-49f8-92be-d66854c43391 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:36.370438 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-142ee634-6cf4-49f8-92be-d66854c43391 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:36.375590 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27de6596-a648-4100-8693-2a598d22486b admin admin] RBAC: Authorizing `identity:create_grant(project_id=0641dd2f31fb4b468e273de5ba93b2bd, user_id=dcf81e7c7942480caf4c6d9d27a5b147, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:36.376353 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-266fb06c-e8d9-4f23-9fe3-e630bb45aea0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:36.376971 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-266fb06c-e8d9-4f23-9fe3-e630bb45aea0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:36.377278 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-266fb06c-e8d9-4f23-9fe3-e630bb45aea0 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:36.377550 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-266fb06c-e8d9-4f23-9fe3-e630bb45aea0 admin admin] PATH_INFO: `/v3/users/a2e772a358e64578850b11fdcf09a403` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:36.378284 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27de6596-a648-4100-8693-2a598d22486b admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:36.378808 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-266fb06c-e8d9-4f23-9fe3-e630bb45aea0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a2e772a358e64578850b11fdcf09a403)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:36.380748 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-266fb06c-e8d9-4f23-9fe3-e630bb45aea0 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:36.391539 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1003/2008] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:53:36 2022] PUT /identity/v3/projects/0641dd2f31fb4b468e273de5ba93b2bd/users/dcf81e7c7942480caf4c6d9d27a5b147/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:36.400431 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=95,keystone:INSERT=4,keystone:DELETE=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:36.402040 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f97b2d-da14-4296-8933-e486624201f5 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:36.402375 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f97b2d-da14-4296-8933-e486624201f5 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:36.402672 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f97b2d-da14-4296-8933-e486624201f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:36.435726 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-266fb06c-e8d9-4f23-9fe3-e630bb45aea0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a2e772a358e64578850b11fdcf09a403'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:36.436565 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-266fb06c-e8d9-4f23-9fe3-e630bb45aea0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a2e772a358e64578850b11fdcf09a403'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:36.437268 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-266fb06c-e8d9-4f23-9fe3-e630bb45aea0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a2e772a358e64578850b11fdcf09a403'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:36.446528 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-27f97b2d-da14-4296-8933-e486624201f5 None None] MFA Rules not processed for user `dcf81e7c7942480caf4c6d9d27a5b147`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:36.447120 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-266fb06c-e8d9-4f23-9fe3-e630bb45aea0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a2e772a358e64578850b11fdcf09a403'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:36.447942 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-27f97b2d-da14-4296-8933-e486624201f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:36.460659 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-266fb06c-e8d9-4f23-9fe3-e630bb45aea0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a2e772a358e64578850b11fdcf09a403'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:36.474103 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1006/2009] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:53:36 2022] DELETE /identity/v3/users/a2e772a358e64578850b11fdcf09a403 => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:53:36.478912 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6acdb60c-87d0-441b-9054-790bc8742140 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:36.479264 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6acdb60c-87d0-441b-9054-790bc8742140 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:36.479607 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6acdb60c-87d0-441b-9054-790bc8742140 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:36.501551 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-6acdb60c-87d0-441b-9054-790bc8742140 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:36.502998 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-6acdb60c-87d0-441b-9054-790bc8742140 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:36.524752 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1004/2010] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:36 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:53:36.536183 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27f97b2d-da14-4296-8933-e486624201f5 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:36.537984 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-27f97b2d-da14-4296-8933-e486624201f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:36.573075 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1007/2011] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:53:36.584469 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6acdb60c-87d0-441b-9054-790bc8742140 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:36.585931 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-6acdb60c-87d0-441b-9054-790bc8742140 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:36.599460 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27f97b2d-da14-4296-8933-e486624201f5 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:36.613178 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:36.654412 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6acdb60c-87d0-441b-9054-790bc8742140 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:36.655431 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-604c9795-c986-4912-9e32-da9c9018a96d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:36.656155 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604c9795-c986-4912-9e32-da9c9018a96d service cinder] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:36.656501 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604c9795-c986-4912-9e32-da9c9018a96d service cinder] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:36.656893 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604c9795-c986-4912-9e32-da9c9018a96d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:36.660214 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-452f2a53-a62e-4b29-9579-3229e1e32406 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:36.660859 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452f2a53-a62e-4b29-9579-3229e1e32406 service neutron] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:36.661137 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452f2a53-a62e-4b29-9579-3229e1e32406 service neutron] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:36.661435 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452f2a53-a62e-4b29-9579-3229e1e32406 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:36.676946 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-452f2a53-a62e-4b29-9579-3229e1e32406 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:36.678389 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-452f2a53-a62e-4b29-9579-3229e1e32406 service neutron] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:36.689447 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1008/2012] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:53:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 11 00:53:36.724676 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-604c9795-c986-4912-9e32-da9c9018a96d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:36.726159 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-604c9795-c986-4912-9e32-da9c9018a96d service cinder] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:36.735908 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1005/2013] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:53:36 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 201 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 11 00:53:37.296284 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-452f2a53-a62e-4b29-9579-3229e1e32406 service neutron] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:37.298766 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-452f2a53-a62e-4b29-9579-3229e1e32406 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:37.318873 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-452f2a53-a62e-4b29-9579-3229e1e32406 service neutron] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:37.324107 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ecda6e3-5a7c-4b3d-a4a9-bab110727ff5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:37.324763 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ecda6e3-5a7c-4b3d-a4a9-bab110727ff5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:37.325042 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ecda6e3-5a7c-4b3d-a4a9-bab110727ff5 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:37.325315 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ecda6e3-5a7c-4b3d-a4a9-bab110727ff5 admin admin] PATH_INFO: `/v3/projects/86b786e1eacd4aceb8e86cee504756e9` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:37.326551 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ecda6e3-5a7c-4b3d-a4a9-bab110727ff5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=86b786e1eacd4aceb8e86cee504756e9)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:37.328502 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ecda6e3-5a7c-4b3d-a4a9-bab110727ff5 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:37.361481 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:37.378475 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-9ecda6e3-5a7c-4b3d-a4a9-bab110727ff5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '86b786e1eacd4aceb8e86cee504756e9', 'request_id': 'req-9ecda6e3-5a7c-4b3d-a4a9-bab110727ff5'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:37.384547 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-9ecda6e3-5a7c-4b3d-a4a9-bab110727ff5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '86b786e1eacd4aceb8e86cee504756e9', 'request_id': 'req-9ecda6e3-5a7c-4b3d-a4a9-bab110727ff5'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:37.390341 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-9ecda6e3-5a7c-4b3d-a4a9-bab110727ff5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '86b786e1eacd4aceb8e86cee504756e9', 'request_id': 'req-9ecda6e3-5a7c-4b3d-a4a9-bab110727ff5'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:37.402467 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-9ecda6e3-5a7c-4b3d-a4a9-bab110727ff5 admin admin] The token cache is being invalidate because project 86b786e1eacd4aceb8e86cee504756e9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87923) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:53:37.402808 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-9ecda6e3-5a7c-4b3d-a4a9-bab110727ff5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:37.403652 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-9ecda6e3-5a7c-4b3d-a4a9-bab110727ff5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:37.405142 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1009/2014] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:53:37 2022] DELETE /identity/v3/projects/86b786e1eacd4aceb8e86cee504756e9 => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:37.409142 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-604c9795-c986-4912-9e32-da9c9018a96d service cinder] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:37.421105 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-604c9795-c986-4912-9e32-da9c9018a96d service cinder] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:37.476075 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e810e42c-75a8-45c3-921f-38802274eb5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:37.476798 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e810e42c-75a8-45c3-921f-38802274eb5e admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:37.476962 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e810e42c-75a8-45c3-921f-38802274eb5e admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:37.477227 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e810e42c-75a8-45c3-921f-38802274eb5e admin admin] PATH_INFO: `/v3/users/38508871925d465cb24e32a83e98fe2b` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:37.478331 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e810e42c-75a8-45c3-921f-38802274eb5e admin admin] RBAC: Authorizing `identity:delete_user(user_id=38508871925d465cb24e32a83e98fe2b)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:37.480270 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e810e42c-75a8-45c3-921f-38802274eb5e admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:37.525012 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-e810e42c-75a8-45c3-921f-38802274eb5e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '38508871925d465cb24e32a83e98fe2b'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:37.525646 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-e810e42c-75a8-45c3-921f-38802274eb5e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '38508871925d465cb24e32a83e98fe2b'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:37.526243 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-e810e42c-75a8-45c3-921f-38802274eb5e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '38508871925d465cb24e32a83e98fe2b'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:37.534045 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-e810e42c-75a8-45c3-921f-38802274eb5e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '38508871925d465cb24e32a83e98fe2b'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:37.546871 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=91,keystone:DELETE=9 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:37.548033 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-e810e42c-75a8-45c3-921f-38802274eb5e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '38508871925d465cb24e32a83e98fe2b'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:37.560627 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1006/2015] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:53:37 2022] DELETE /identity/v3/users/38508871925d465cb24e32a83e98fe2b => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:37.906232 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ecda6e3-5a7c-4b3d-a4a9-bab110727ff5 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:37.907829 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-9ecda6e3-5a7c-4b3d-a4a9-bab110727ff5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:37.927219 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ecda6e3-5a7c-4b3d-a4a9-bab110727ff5 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:37.991555 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac370ed-c8b6-4e4a-8f98-924f8c5a879a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:37.992209 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac370ed-c8b6-4e4a-8f98-924f8c5a879a admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:37.992485 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac370ed-c8b6-4e4a-8f98-924f8c5a879a admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:37.992752 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac370ed-c8b6-4e4a-8f98-924f8c5a879a admin admin] PATH_INFO: `/v3/projects/25b5c52f70104623ac36dbcd0c20bab5` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:37.993866 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ac370ed-c8b6-4e4a-8f98-924f8c5a879a admin admin] RBAC: Authorizing `identity:delete_project(project_id=25b5c52f70104623ac36dbcd0c20bab5)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:37.995810 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ac370ed-c8b6-4e4a-8f98-924f8c5a879a admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:38.041736 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-3ac370ed-c8b6-4e4a-8f98-924f8c5a879a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '25b5c52f70104623ac36dbcd0c20bab5', 'request_id': 'req-3ac370ed-c8b6-4e4a-8f98-924f8c5a879a'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:38.048293 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-3ac370ed-c8b6-4e4a-8f98-924f8c5a879a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '25b5c52f70104623ac36dbcd0c20bab5', 'request_id': 'req-3ac370ed-c8b6-4e4a-8f98-924f8c5a879a'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:38.053794 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-3ac370ed-c8b6-4e4a-8f98-924f8c5a879a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '25b5c52f70104623ac36dbcd0c20bab5', 'request_id': 'req-3ac370ed-c8b6-4e4a-8f98-924f8c5a879a'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:38.073245 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-3ac370ed-c8b6-4e4a-8f98-924f8c5a879a admin admin] The token cache is being invalidate because project 25b5c52f70104623ac36dbcd0c20bab5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87923) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:53:38.073459 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-3ac370ed-c8b6-4e4a-8f98-924f8c5a879a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:38.074093 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-3ac370ed-c8b6-4e4a-8f98-924f8c5a879a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:38.075751 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1010/2016] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:53:37 2022] DELETE /identity/v3/projects/25b5c52f70104623ac36dbcd0c20bab5 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:38.080645 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae58dd42-3f00-47f7-9632-4db55b2f6834 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:38.080984 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae58dd42-3f00-47f7-9632-4db55b2f6834 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:38.081262 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae58dd42-3f00-47f7-9632-4db55b2f6834 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:38.101776 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-ae58dd42-3f00-47f7-9632-4db55b2f6834 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:38.103174 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-ae58dd42-3f00-47f7-9632-4db55b2f6834 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:38.167796 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1007/2017] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:53:38.176031 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f402eb1-cdf0-4c87-b332-f5912e2f1e9d None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:38.176323 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f402eb1-cdf0-4c87-b332-f5912e2f1e9d None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:38.176601 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f402eb1-cdf0-4c87-b332-f5912e2f1e9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:38.196229 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-8f402eb1-cdf0-4c87-b332-f5912e2f1e9d None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:38.197419 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-8f402eb1-cdf0-4c87-b332-f5912e2f1e9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:38.206102 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1011/2018] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:53:38.210303 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae58dd42-3f00-47f7-9632-4db55b2f6834 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:38.220745 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae58dd42-3f00-47f7-9632-4db55b2f6834 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:38.225340 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2acacf80-4420-4d46-b745-f6c9d397dd32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:38.225938 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2acacf80-4420-4d46-b745-f6c9d397dd32 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:38.226221 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2acacf80-4420-4d46-b745-f6c9d397dd32 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:38.226494 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2acacf80-4420-4d46-b745-f6c9d397dd32 admin admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:38.227239 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2acacf80-4420-4d46-b745-f6c9d397dd32 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:53:38.227544 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2acacf80-4420-4d46-b745-f6c9d397dd32 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:38.229127 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2acacf80-4420-4d46-b745-f6c9d397dd32 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:38.240604 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1008/2019] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 00:53:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:53:38.245951 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f402eb1-cdf0-4c87-b332-f5912e2f1e9d None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:38.255659 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f402eb1-cdf0-4c87-b332-f5912e2f1e9d None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:38.260269 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec01483-f1e7-4354-8988-9fef964f18ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:38.260860 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec01483-f1e7-4354-8988-9fef964f18ed admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:38.261137 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec01483-f1e7-4354-8988-9fef964f18ed admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:38.261412 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec01483-f1e7-4354-8988-9fef964f18ed admin admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:38.262104 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dec01483-f1e7-4354-8988-9fef964f18ed admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:38.264847 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dec01483-f1e7-4354-8988-9fef964f18ed admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:38.291504 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1012/2020] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:53:38 2022] POST /identity/v3/projects => generated 407 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:53:38.295708 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2acacf80-4420-4d46-b745-f6c9d397dd32 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:38.305591 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2acacf80-4420-4d46-b745-f6c9d397dd32 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:38.365844 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a82bcb-58b4-4f08-8b49-bd900354b26f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:38.366454 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a82bcb-58b4-4f08-8b49-bd900354b26f admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:38.366732 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a82bcb-58b4-4f08-8b49-bd900354b26f admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:38.367035 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a82bcb-58b4-4f08-8b49-bd900354b26f admin admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:38.367665 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04a82bcb-58b4-4f08-8b49-bd900354b26f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:38.369604 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04a82bcb-58b4-4f08-8b49-bd900354b26f admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:38.391926 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1009/2021] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:53:38 2022] POST /identity/v3/users => generated 304 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:53:38.395879 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec01483-f1e7-4354-8988-9fef964f18ed admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:38.406483 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec01483-f1e7-4354-8988-9fef964f18ed admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:38.411566 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a9264f0-48fb-49f4-b060-d1b6ae9bd06c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:38.412261 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a9264f0-48fb-49f4-b060-d1b6ae9bd06c admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:38.412616 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a9264f0-48fb-49f4-b060-d1b6ae9bd06c admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:38.412951 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a9264f0-48fb-49f4-b060-d1b6ae9bd06c admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:38.413663 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a9264f0-48fb-49f4-b060-d1b6ae9bd06c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:38.415304 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a9264f0-48fb-49f4-b060-d1b6ae9bd06c admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:38.425837 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1013/2022] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:53:38 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 11 00:53:38.429818 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a82bcb-58b4-4f08-8b49-bd900354b26f admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:38.441109 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a82bcb-58b4-4f08-8b49-bd900354b26f admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:38.446948 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-932cd8d6-d6ef-4d61-84e3-587ad44b55f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:38.447793 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932cd8d6-d6ef-4d61-84e3-587ad44b55f7 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:38.448074 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932cd8d6-d6ef-4d61-84e3-587ad44b55f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:38.448958 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932cd8d6-d6ef-4d61-84e3-587ad44b55f7 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:38.450282 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-932cd8d6-d6ef-4d61-84e3-587ad44b55f7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:38.452160 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-932cd8d6-d6ef-4d61-84e3-587ad44b55f7 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:38.463487 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1010/2023] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:53:38 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:53:38.467524 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a9264f0-48fb-49f4-b060-d1b6ae9bd06c admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:38.477594 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a9264f0-48fb-49f4-b060-d1b6ae9bd06c admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:38.482662 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6867b66-c87f-4cb6-a9db-6e9308496ec4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:38.483382 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6867b66-c87f-4cb6-a9db-6e9308496ec4 admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:38.483663 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6867b66-c87f-4cb6-a9db-6e9308496ec4 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:38.483928 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6867b66-c87f-4cb6-a9db-6e9308496ec4 admin admin] PATH_INFO: `/v3/projects/392e314f424741e28fec3ba6aa0c7efb/users/fca18805a0d64bf69766aa14ed75853a/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:38.486354 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=85,keystone:DELETE=12,keystone:INSERT=3 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:38.501501 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6867b66-c87f-4cb6-a9db-6e9308496ec4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=392e314f424741e28fec3ba6aa0c7efb, user_id=fca18805a0d64bf69766aa14ed75853a, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:38.504166 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6867b66-c87f-4cb6-a9db-6e9308496ec4 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:38.517295 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1014/2024] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:53:38 2022] PUT /identity/v3/projects/392e314f424741e28fec3ba6aa0c7efb/users/fca18805a0d64bf69766aa14ed75853a/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:38.529084 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0974bc8-70b5-4a5a-9eaa-426d5c7000a2 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:38.529582 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0974bc8-70b5-4a5a-9eaa-426d5c7000a2 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:38.529677 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0974bc8-70b5-4a5a-9eaa-426d5c7000a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:38.558590 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-f0974bc8-70b5-4a5a-9eaa-426d5c7000a2 None None] MFA Rules not processed for user `fca18805a0d64bf69766aa14ed75853a`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:38.559901 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-f0974bc8-70b5-4a5a-9eaa-426d5c7000a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:38.605270 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=95,keystone:INSERT=4,keystone:DELETE=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:38.627275 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1011/2025] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:38 2022] POST /identity/v3/auth/tokens => generated 3284 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:53:38.631905 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6867b66-c87f-4cb6-a9db-6e9308496ec4 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:38.693896 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6867b66-c87f-4cb6-a9db-6e9308496ec4 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:38.698515 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab12fb4-8814-4537-99e6-fe1f5fe6b4af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:38.699154 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab12fb4-8814-4537-99e6-fe1f5fe6b4af admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:38.699432 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab12fb4-8814-4537-99e6-fe1f5fe6b4af admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:38.700353 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab12fb4-8814-4537-99e6-fe1f5fe6b4af admin admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:38.701044 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ab12fb4-8814-4537-99e6-fe1f5fe6b4af admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:38.702959 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ab12fb4-8814-4537-99e6-fe1f5fe6b4af admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:38.730918 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1015/2026] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:53:38 2022] POST /identity/v3/projects => generated 409 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:53:38.735189 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0974bc8-70b5-4a5a-9eaa-426d5c7000a2 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:38.747700 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0974bc8-70b5-4a5a-9eaa-426d5c7000a2 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:38.808542 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67b83d78-6092-4e37-9396-b984a457c84f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:38.809186 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b83d78-6092-4e37-9396-b984a457c84f admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:38.809455 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b83d78-6092-4e37-9396-b984a457c84f admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:38.809792 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b83d78-6092-4e37-9396-b984a457c84f admin admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:38.810468 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67b83d78-6092-4e37-9396-b984a457c84f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:38.812840 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67b83d78-6092-4e37-9396-b984a457c84f admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:38.836843 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1012/2027] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:53:38 2022] POST /identity/v3/users => generated 311 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:53:38.840139 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab12fb4-8814-4537-99e6-fe1f5fe6b4af admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:38.852113 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab12fb4-8814-4537-99e6-fe1f5fe6b4af admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:38.858561 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26571569-d823-48ad-b925-0be0adc1d9be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:38.859384 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26571569-d823-48ad-b925-0be0adc1d9be admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:38.859688 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26571569-d823-48ad-b925-0be0adc1d9be admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:38.859974 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26571569-d823-48ad-b925-0be0adc1d9be admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:38.860722 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26571569-d823-48ad-b925-0be0adc1d9be admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:38.866993 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26571569-d823-48ad-b925-0be0adc1d9be admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:38.886850 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1016/2028] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:53:38 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:53:38.891417 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67b83d78-6092-4e37-9396-b984a457c84f admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:38.905194 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67b83d78-6092-4e37-9396-b984a457c84f admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:38.911865 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c8cbcb-ea66-456f-8b9c-538598a9cd24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:38.912640 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c8cbcb-ea66-456f-8b9c-538598a9cd24 admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:38.912989 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c8cbcb-ea66-456f-8b9c-538598a9cd24 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:38.913663 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c8cbcb-ea66-456f-8b9c-538598a9cd24 admin admin] PATH_INFO: `/v3/projects/383dadd676cb4a7cbdfbf40edce8df3e/users/a41b64c71668411e90e980b92edb8a90/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:38.936433 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c8cbcb-ea66-456f-8b9c-538598a9cd24 admin admin] RBAC: Authorizing `identity:create_grant(project_id=383dadd676cb4a7cbdfbf40edce8df3e, user_id=a41b64c71668411e90e980b92edb8a90, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:38.936433 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c8cbcb-ea66-456f-8b9c-538598a9cd24 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:38.969973 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26571569-d823-48ad-b925-0be0adc1d9be admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:38.970154 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1013/2029] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:53:38 2022] PUT /identity/v3/projects/383dadd676cb4a7cbdfbf40edce8df3e/users/a41b64c71668411e90e980b92edb8a90/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 73 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:53:38.983112 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26571569-d823-48ad-b925-0be0adc1d9be admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:39.087382 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e947e14-a97e-478d-a2c1-6edee3adbf8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:39.088161 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e947e14-a97e-478d-a2c1-6edee3adbf8d admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:39.088532 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e947e14-a97e-478d-a2c1-6edee3adbf8d admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:39.088896 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e947e14-a97e-478d-a2c1-6edee3adbf8d admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:39.091111 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e947e14-a97e-478d-a2c1-6edee3adbf8d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:39.093201 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e947e14-a97e-478d-a2c1-6edee3adbf8d admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:39.112053 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1017/2030] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:53:38 2022] GET /identity/v3/roles => generated 1420 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:53:39.116234 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c8cbcb-ea66-456f-8b9c-538598a9cd24 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:39.127843 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c8cbcb-ea66-456f-8b9c-538598a9cd24 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:39.132831 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c1f71b3-b513-4d82-b07d-585e3ef8b52f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:39.133495 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1f71b3-b513-4d82-b07d-585e3ef8b52f admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:39.133904 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1f71b3-b513-4d82-b07d-585e3ef8b52f admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:39.134279 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1f71b3-b513-4d82-b07d-585e3ef8b52f admin admin] PATH_INFO: `/v3/projects/383dadd676cb4a7cbdfbf40edce8df3e/users/a41b64c71668411e90e980b92edb8a90/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:39.136379 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c1f71b3-b513-4d82-b07d-585e3ef8b52f admin admin] RBAC: Authorizing `identity:create_grant(project_id=383dadd676cb4a7cbdfbf40edce8df3e, user_id=a41b64c71668411e90e980b92edb8a90, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:39.139901 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c1f71b3-b513-4d82-b07d-585e3ef8b52f admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:39.159348 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1014/2031] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:53:39 2022] PUT /identity/v3/projects/383dadd676cb4a7cbdfbf40edce8df3e/users/a41b64c71668411e90e980b92edb8a90/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 44 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:39.171180 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e6dddd-e781-4409-8c5f-c100ae089168 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:39.171597 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e6dddd-e781-4409-8c5f-c100ae089168 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:39.172100 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e6dddd-e781-4409-8c5f-c100ae089168 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:39.212120 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-c6e6dddd-e781-4409-8c5f-c100ae089168 None None] MFA Rules not processed for user `a41b64c71668411e90e980b92edb8a90`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:39.213904 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-c6e6dddd-e781-4409-8c5f-c100ae089168 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:39.271071 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:39.303690 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1018/2032] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:39 2022] POST /identity/v3/auth/tokens => generated 3353 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:53:39.317100 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c1f71b3-b513-4d82-b07d-585e3ef8b52f admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:39.319372 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-2c1f71b3-b513-4d82-b07d-585e3ef8b52f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:39.393031 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c1f71b3-b513-4d82-b07d-585e3ef8b52f admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:39.396729 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf82aa16-8f89-4113-8833-17ed975b09d1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:39.397353 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf82aa16-8f89-4113-8833-17ed975b09d1 service cinder] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:39.397633 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf82aa16-8f89-4113-8833-17ed975b09d1 service cinder] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:39.397901 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf82aa16-8f89-4113-8833-17ed975b09d1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:39.408338 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf82aa16-8f89-4113-8833-17ed975b09d1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:39.410774 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf82aa16-8f89-4113-8833-17ed975b09d1 service cinder] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:39.423281 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1015/2033] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:53:39 2022] GET /identity/v3/auth/tokens => generated 3353 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:53:39.603530 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6e6dddd-e781-4409-8c5f-c100ae089168 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:39.606135 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-c6e6dddd-e781-4409-8c5f-c100ae089168 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:39.622117 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6e6dddd-e781-4409-8c5f-c100ae089168 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:39.626476 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-622a045e-e800-487f-b6aa-bbe02c00991b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:39.627250 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622a045e-e800-487f-b6aa-bbe02c00991b service cinder] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:39.627393 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622a045e-e800-487f-b6aa-bbe02c00991b service cinder] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:39.627665 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622a045e-e800-487f-b6aa-bbe02c00991b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:39.696736 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-622a045e-e800-487f-b6aa-bbe02c00991b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:39.698047 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-622a045e-e800-487f-b6aa-bbe02c00991b service cinder] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:39.707583 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1019/2034] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:53:39 2022] GET /identity/v3/auth/tokens => generated 3284 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:53:48.438338 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf82aa16-8f89-4113-8833-17ed975b09d1 service cinder] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:48.439793 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-bf82aa16-8f89-4113-8833-17ed975b09d1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:48.482961 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:48.505857 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf82aa16-8f89-4113-8833-17ed975b09d1 service cinder] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:48.510299 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d973e19-dc98-457e-999f-c88e7b553af9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:48.510952 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d973e19-dc98-457e-999f-c88e7b553af9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:48.511229 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d973e19-dc98-457e-999f-c88e7b553af9 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:48.511496 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d973e19-dc98-457e-999f-c88e7b553af9 admin admin] PATH_INFO: `/v3/users/dcf81e7c7942480caf4c6d9d27a5b147` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:48.513304 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d973e19-dc98-457e-999f-c88e7b553af9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=dcf81e7c7942480caf4c6d9d27a5b147)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:48.515232 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d973e19-dc98-457e-999f-c88e7b553af9 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:48.573027 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-2d973e19-dc98-457e-999f-c88e7b553af9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dcf81e7c7942480caf4c6d9d27a5b147'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:48.573678 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-2d973e19-dc98-457e-999f-c88e7b553af9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dcf81e7c7942480caf4c6d9d27a5b147'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:48.574968 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-2d973e19-dc98-457e-999f-c88e7b553af9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dcf81e7c7942480caf4c6d9d27a5b147'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:48.582921 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-2d973e19-dc98-457e-999f-c88e7b553af9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dcf81e7c7942480caf4c6d9d27a5b147'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:48.596949 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-2d973e19-dc98-457e-999f-c88e7b553af9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dcf81e7c7942480caf4c6d9d27a5b147'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:48.609965 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1016/2035] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:53:48 2022] DELETE /identity/v3/users/dcf81e7c7942480caf4c6d9d27a5b147 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:48.614562 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea780ab7-78bf-45fb-bf91-0b92247570cc None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:48.615109 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea780ab7-78bf-45fb-bf91-0b92247570cc None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:48.615335 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea780ab7-78bf-45fb-bf91-0b92247570cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:48.635553 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-ea780ab7-78bf-45fb-bf91-0b92247570cc None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:48.637615 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-ea780ab7-78bf-45fb-bf91-0b92247570cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:48.706454 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1020/2036] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:53:48.716808 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d973e19-dc98-457e-999f-c88e7b553af9 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:48.718269 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-2d973e19-dc98-457e-999f-c88e7b553af9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:48.782535 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d973e19-dc98-457e-999f-c88e7b553af9 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:48.787241 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef07d48-a647-4c32-91eb-138094efffbc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:48.787861 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef07d48-a647-4c32-91eb-138094efffbc service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:48.788270 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef07d48-a647-4c32-91eb-138094efffbc service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:48.788580 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef07d48-a647-4c32-91eb-138094efffbc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:48.798313 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ef07d48-a647-4c32-91eb-138094efffbc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:48.799781 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ef07d48-a647-4c32-91eb-138094efffbc service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:48.811337 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1017/2037] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:53:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:53:49.134590 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea780ab7-78bf-45fb-bf91-0b92247570cc None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:49.136063 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-ea780ab7-78bf-45fb-bf91-0b92247570cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:49.149817 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea780ab7-78bf-45fb-bf91-0b92247570cc None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:49.155390 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5614ff-a14a-45ae-a9c8-95406df7218c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:49.156056 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5614ff-a14a-45ae-a9c8-95406df7218c admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:49.156343 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5614ff-a14a-45ae-a9c8-95406df7218c admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:49.156615 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5614ff-a14a-45ae-a9c8-95406df7218c admin admin] PATH_INFO: `/v3/projects/0641dd2f31fb4b468e273de5ba93b2bd` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:49.157859 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e5614ff-a14a-45ae-a9c8-95406df7218c admin admin] RBAC: Authorizing `identity:delete_project(project_id=0641dd2f31fb4b468e273de5ba93b2bd)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:49.159823 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e5614ff-a14a-45ae-a9c8-95406df7218c admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:49.208062 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-4e5614ff-a14a-45ae-a9c8-95406df7218c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0641dd2f31fb4b468e273de5ba93b2bd', 'request_id': 'req-4e5614ff-a14a-45ae-a9c8-95406df7218c'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:49.214239 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-4e5614ff-a14a-45ae-a9c8-95406df7218c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0641dd2f31fb4b468e273de5ba93b2bd', 'request_id': 'req-4e5614ff-a14a-45ae-a9c8-95406df7218c'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:49.220827 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-4e5614ff-a14a-45ae-a9c8-95406df7218c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0641dd2f31fb4b468e273de5ba93b2bd', 'request_id': 'req-4e5614ff-a14a-45ae-a9c8-95406df7218c'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:49.229402 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:49.233525 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-4e5614ff-a14a-45ae-a9c8-95406df7218c admin admin] The token cache is being invalidate because project 0641dd2f31fb4b468e273de5ba93b2bd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87923) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:53:49.234169 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-4e5614ff-a14a-45ae-a9c8-95406df7218c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:49.234860 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-4e5614ff-a14a-45ae-a9c8-95406df7218c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:49.236427 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1021/2038] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:53:49 2022] DELETE /identity/v3/projects/0641dd2f31fb4b468e273de5ba93b2bd => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:49.241294 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cfa2f31-9ffe-4892-97f8-4e735870541d None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:49.241573 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cfa2f31-9ffe-4892-97f8-4e735870541d None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:49.241842 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cfa2f31-9ffe-4892-97f8-4e735870541d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:49.261166 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-1cfa2f31-9ffe-4892-97f8-4e735870541d None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:49.263147 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-1cfa2f31-9ffe-4892-97f8-4e735870541d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:49.321701 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1018/2039] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:53:49.330680 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d63439-1ad4-4c0b-872d-a8490a17d55c None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:49.331032 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d63439-1ad4-4c0b-872d-a8490a17d55c None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:49.331305 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d63439-1ad4-4c0b-872d-a8490a17d55c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:49.351237 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-c1d63439-1ad4-4c0b-872d-a8490a17d55c None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:49.352463 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-c1d63439-1ad4-4c0b-872d-a8490a17d55c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:49.361914 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1022/2040] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:53:49.366404 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cfa2f31-9ffe-4892-97f8-4e735870541d None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:49.372505 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:49.380414 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cfa2f31-9ffe-4892-97f8-4e735870541d None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:49.386490 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0ccd3c-5862-4162-a058-ed3698df6323 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:49.387412 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0ccd3c-5862-4162-a058-ed3698df6323 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:49.387812 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0ccd3c-5862-4162-a058-ed3698df6323 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:49.388085 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0ccd3c-5862-4162-a058-ed3698df6323 admin admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:49.388735 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e0ccd3c-5862-4162-a058-ed3698df6323 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:53:49.389035 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e0ccd3c-5862-4162-a058-ed3698df6323 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:49.390626 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e0ccd3c-5862-4162-a058-ed3698df6323 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:49.402269 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1019/2041] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 00:53:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:53:49.406687 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d63439-1ad4-4c0b-872d-a8490a17d55c None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:49.416288 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d63439-1ad4-4c0b-872d-a8490a17d55c None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:49.421478 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1bc0988-5ca4-40c1-a816-0fc8ec0fa206 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:49.422074 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1bc0988-5ca4-40c1-a816-0fc8ec0fa206 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:49.422350 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1bc0988-5ca4-40c1-a816-0fc8ec0fa206 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:49.422618 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1bc0988-5ca4-40c1-a816-0fc8ec0fa206 admin admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:49.423273 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1bc0988-5ca4-40c1-a816-0fc8ec0fa206 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:49.425166 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1bc0988-5ca4-40c1-a816-0fc8ec0fa206 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:49.452391 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1023/2042] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:53:49 2022] POST /identity/v3/projects => generated 373 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:53:49.456553 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0ccd3c-5862-4162-a058-ed3698df6323 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:49.467341 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0ccd3c-5862-4162-a058-ed3698df6323 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:49.522286 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2cc845c-e6df-4332-b290-6c6d0ae2120c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:49.523282 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2cc845c-e6df-4332-b290-6c6d0ae2120c admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:49.523547 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2cc845c-e6df-4332-b290-6c6d0ae2120c admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:49.523815 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2cc845c-e6df-4332-b290-6c6d0ae2120c admin admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:49.524443 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2cc845c-e6df-4332-b290-6c6d0ae2120c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:49.526346 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2cc845c-e6df-4332-b290-6c6d0ae2120c admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:49.549580 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1020/2043] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:53:49 2022] POST /identity/v3/users => generated 287 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:53:49.553734 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1bc0988-5ca4-40c1-a816-0fc8ec0fa206 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:49.564079 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1bc0988-5ca4-40c1-a816-0fc8ec0fa206 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:49.568729 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb33a922-3e8a-42b5-8ca5-77ee57f2a143 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:49.569330 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb33a922-3e8a-42b5-8ca5-77ee57f2a143 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:49.569607 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb33a922-3e8a-42b5-8ca5-77ee57f2a143 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:49.569879 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb33a922-3e8a-42b5-8ca5-77ee57f2a143 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:49.570525 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb33a922-3e8a-42b5-8ca5-77ee57f2a143 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:49.572253 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb33a922-3e8a-42b5-8ca5-77ee57f2a143 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:49.582505 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1024/2044] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:53:49 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:53:49.586255 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2cc845c-e6df-4332-b290-6c6d0ae2120c admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:49.597004 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2cc845c-e6df-4332-b290-6c6d0ae2120c admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:49.601688 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2021d451-fde4-4694-87ad-f816522b32c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:49.602280 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2021d451-fde4-4694-87ad-f816522b32c9 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:49.602557 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2021d451-fde4-4694-87ad-f816522b32c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:49.602870 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2021d451-fde4-4694-87ad-f816522b32c9 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:49.603511 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2021d451-fde4-4694-87ad-f816522b32c9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:49.605092 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2021d451-fde4-4694-87ad-f816522b32c9 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:49.616342 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1021/2045] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:53:49 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:53:49.620440 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb33a922-3e8a-42b5-8ca5-77ee57f2a143 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:49.630404 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb33a922-3e8a-42b5-8ca5-77ee57f2a143 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:49.635134 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c2774ac-c8a3-4605-9afa-71067ff4b579 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:49.635758 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2774ac-c8a3-4605-9afa-71067ff4b579 admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:49.636038 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2774ac-c8a3-4605-9afa-71067ff4b579 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:49.636310 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2774ac-c8a3-4605-9afa-71067ff4b579 admin admin] PATH_INFO: `/v3/projects/b6155e3b936b4675bfa9315475f66342/users/8f3d3f1dc33d414c9afeb2233d7d6e4a/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:49.651188 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c2774ac-c8a3-4605-9afa-71067ff4b579 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b6155e3b936b4675bfa9315475f66342, user_id=8f3d3f1dc33d414c9afeb2233d7d6e4a, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:49.654384 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c2774ac-c8a3-4605-9afa-71067ff4b579 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:49.667399 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1025/2046] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:53:49 2022] PUT /identity/v3/projects/b6155e3b936b4675bfa9315475f66342/users/8f3d3f1dc33d414c9afeb2233d7d6e4a/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:49.679589 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38ef7d1d-fa20-4124-bba7-8bebb4e19ae6 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:49.680216 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38ef7d1d-fa20-4124-bba7-8bebb4e19ae6 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:49.680216 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38ef7d1d-fa20-4124-bba7-8bebb4e19ae6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:49.708353 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-38ef7d1d-fa20-4124-bba7-8bebb4e19ae6 None None] MFA Rules not processed for user `8f3d3f1dc33d414c9afeb2233d7d6e4a`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:49.709576 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-38ef7d1d-fa20-4124-bba7-8bebb4e19ae6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:49.778122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1022/2047] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:49 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:53:49.789239 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c2774ac-c8a3-4605-9afa-71067ff4b579 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:49.791004 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-9c2774ac-c8a3-4605-9afa-71067ff4b579 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:49.850717 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c2774ac-c8a3-4605-9afa-71067ff4b579 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:49.854330 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954ae93b-df21-42c5-b079-642beebab6e7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:49.854960 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954ae93b-df21-42c5-b079-642beebab6e7 service cinder] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:49.855237 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954ae93b-df21-42c5-b079-642beebab6e7 service cinder] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:49.855645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954ae93b-df21-42c5-b079-642beebab6e7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:49.864159 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-954ae93b-df21-42c5-b079-642beebab6e7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:49.865596 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-954ae93b-df21-42c5-b079-642beebab6e7 service cinder] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:49.874396 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1026/2048] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:53:49 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:53:56.644043 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38ef7d1d-fa20-4124-bba7-8bebb4e19ae6 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:56.645513 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-38ef7d1d-fa20-4124-bba7-8bebb4e19ae6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:56.706824 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:56.711315 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38ef7d1d-fa20-4124-bba7-8bebb4e19ae6 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:56.717175 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d26cf42b-7d0d-419d-b4c0-7f734747ecd1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:56.717804 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d26cf42b-7d0d-419d-b4c0-7f734747ecd1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:56.718199 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d26cf42b-7d0d-419d-b4c0-7f734747ecd1 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:56.718478 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d26cf42b-7d0d-419d-b4c0-7f734747ecd1 admin admin] PATH_INFO: `/v3/users/fca18805a0d64bf69766aa14ed75853a` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:56.719737 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d26cf42b-7d0d-419d-b4c0-7f734747ecd1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fca18805a0d64bf69766aa14ed75853a)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:56.721734 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d26cf42b-7d0d-419d-b4c0-7f734747ecd1 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:56.769790 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-d26cf42b-7d0d-419d-b4c0-7f734747ecd1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fca18805a0d64bf69766aa14ed75853a'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:56.770443 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-d26cf42b-7d0d-419d-b4c0-7f734747ecd1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fca18805a0d64bf69766aa14ed75853a'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:56.771069 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-d26cf42b-7d0d-419d-b4c0-7f734747ecd1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fca18805a0d64bf69766aa14ed75853a'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:56.778196 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-d26cf42b-7d0d-419d-b4c0-7f734747ecd1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fca18805a0d64bf69766aa14ed75853a'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:56.790525 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-d26cf42b-7d0d-419d-b4c0-7f734747ecd1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fca18805a0d64bf69766aa14ed75853a'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:56.805692 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1023/2049] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:53:56 2022] DELETE /identity/v3/users/fca18805a0d64bf69766aa14ed75853a => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:56.810863 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c85e4f8-1d7d-4767-946c-fca57345ed1b None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:56.811515 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c85e4f8-1d7d-4767-946c-fca57345ed1b None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:56.811617 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c85e4f8-1d7d-4767-946c-fca57345ed1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:56.832168 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-0c85e4f8-1d7d-4767-946c-fca57345ed1b None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:56.833475 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-0c85e4f8-1d7d-4767-946c-fca57345ed1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:56.895596 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1027/2050] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:53:56.905902 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d26cf42b-7d0d-419d-b4c0-7f734747ecd1 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:56.907517 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-d26cf42b-7d0d-419d-b4c0-7f734747ecd1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:56.970347 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d26cf42b-7d0d-419d-b4c0-7f734747ecd1 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:56.974012 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-159e2fbc-88d8-466f-8ec1-22d2bf919886 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:56.974627 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159e2fbc-88d8-466f-8ec1-22d2bf919886 service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:56.974945 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159e2fbc-88d8-466f-8ec1-22d2bf919886 service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:56.975216 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159e2fbc-88d8-466f-8ec1-22d2bf919886 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:56.984381 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-159e2fbc-88d8-466f-8ec1-22d2bf919886 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:56.985808 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-159e2fbc-88d8-466f-8ec1-22d2bf919886 service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:56.995344 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1024/2051] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:53:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:53:57.315943 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c85e4f8-1d7d-4767-946c-fca57345ed1b None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:57.318244 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-0c85e4f8-1d7d-4767-946c-fca57345ed1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:57.332727 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c85e4f8-1d7d-4767-946c-fca57345ed1b None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:57.340039 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659391b1-d2a9-4a4f-9aae-20d69eaa7922 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:57.340851 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659391b1-d2a9-4a4f-9aae-20d69eaa7922 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:57.341288 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659391b1-d2a9-4a4f-9aae-20d69eaa7922 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:57.341665 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659391b1-d2a9-4a4f-9aae-20d69eaa7922 admin admin] PATH_INFO: `/v3/projects/392e314f424741e28fec3ba6aa0c7efb` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:57.342985 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659391b1-d2a9-4a4f-9aae-20d69eaa7922 admin admin] RBAC: Authorizing `identity:delete_project(project_id=392e314f424741e28fec3ba6aa0c7efb)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:57.342985 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659391b1-d2a9-4a4f-9aae-20d69eaa7922 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:57.347229 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-159e2fbc-88d8-466f-8ec1-22d2bf919886 service neutron] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:57.349369 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-159e2fbc-88d8-466f-8ec1-22d2bf919886 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:57.365590 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-159e2fbc-88d8-466f-8ec1-22d2bf919886 service neutron] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:57.373803 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17ccba44-26a4-4629-b148-c0563794960b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:57.374682 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ccba44-26a4-4629-b148-c0563794960b admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:57.375852 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ccba44-26a4-4629-b148-c0563794960b admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:57.376748 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ccba44-26a4-4629-b148-c0563794960b admin admin] PATH_INFO: `/v3/users/8f3d3f1dc33d414c9afeb2233d7d6e4a` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:57.378459 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17ccba44-26a4-4629-b148-c0563794960b admin admin] RBAC: Authorizing `identity:delete_user(user_id=8f3d3f1dc33d414c9afeb2233d7d6e4a)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:57.381485 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17ccba44-26a4-4629-b148-c0563794960b admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:57.400317 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=96,keystone:INSERT=2,keystone:DELETE=2 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:57.420033 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-659391b1-d2a9-4a4f-9aae-20d69eaa7922 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '392e314f424741e28fec3ba6aa0c7efb', 'request_id': 'req-659391b1-d2a9-4a4f-9aae-20d69eaa7922'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:57.430363 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-659391b1-d2a9-4a4f-9aae-20d69eaa7922 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '392e314f424741e28fec3ba6aa0c7efb', 'request_id': 'req-659391b1-d2a9-4a4f-9aae-20d69eaa7922'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:57.450948 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-659391b1-d2a9-4a4f-9aae-20d69eaa7922 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '392e314f424741e28fec3ba6aa0c7efb', 'request_id': 'req-659391b1-d2a9-4a4f-9aae-20d69eaa7922'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:57.462704 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-17ccba44-26a4-4629-b148-c0563794960b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8f3d3f1dc33d414c9afeb2233d7d6e4a'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:57.464485 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-17ccba44-26a4-4629-b148-c0563794960b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8f3d3f1dc33d414c9afeb2233d7d6e4a'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:57.465832 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-17ccba44-26a4-4629-b148-c0563794960b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8f3d3f1dc33d414c9afeb2233d7d6e4a'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:57.474633 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-659391b1-d2a9-4a4f-9aae-20d69eaa7922 admin admin] The token cache is being invalidate because project 392e314f424741e28fec3ba6aa0c7efb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87923) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:53:57.475176 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-17ccba44-26a4-4629-b148-c0563794960b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8f3d3f1dc33d414c9afeb2233d7d6e4a'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:57.475531 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-659391b1-d2a9-4a4f-9aae-20d69eaa7922 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:57.476409 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-659391b1-d2a9-4a4f-9aae-20d69eaa7922 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:57.478691 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1028/2052] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:53:57 2022] DELETE /identity/v3/projects/392e314f424741e28fec3ba6aa0c7efb => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:57.486234 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659391b1-d2a9-4a4f-9aae-20d69eaa7922 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:57.489461 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-659391b1-d2a9-4a4f-9aae-20d69eaa7922 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:57.505957 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-17ccba44-26a4-4629-b148-c0563794960b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8f3d3f1dc33d414c9afeb2233d7d6e4a'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:57.508361 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=80,keystone:DELETE=19,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:57.535640 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1025/2053] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:53:57 2022] DELETE /identity/v3/users/8f3d3f1dc33d414c9afeb2233d7d6e4a => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:53:57.540866 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4fae86-771d-42c3-a709-1eec990dc33f None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:57.541427 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4fae86-771d-42c3-a709-1eec990dc33f None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:57.541698 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4fae86-771d-42c3-a709-1eec990dc33f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:57.567637 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-1d4fae86-771d-42c3-a709-1eec990dc33f None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:57.569156 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-1d4fae86-771d-42c3-a709-1eec990dc33f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:57.634883 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659391b1-d2a9-4a4f-9aae-20d69eaa7922 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:57.681352 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1026/2054] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:53:57.691951 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d4fae86-771d-42c3-a709-1eec990dc33f None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:57.693423 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-1d4fae86-771d-42c3-a709-1eec990dc33f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:57.755047 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fdd97b6-d458-48c9-b7bb-06569cc6bf00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:57.755726 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fdd97b6-d458-48c9-b7bb-06569cc6bf00 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:57.756011 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fdd97b6-d458-48c9-b7bb-06569cc6bf00 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:57.756309 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fdd97b6-d458-48c9-b7bb-06569cc6bf00 admin admin] PATH_INFO: `/v3/users/a41b64c71668411e90e980b92edb8a90` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:57.757949 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fdd97b6-d458-48c9-b7bb-06569cc6bf00 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a41b64c71668411e90e980b92edb8a90)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:57.760111 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fdd97b6-d458-48c9-b7bb-06569cc6bf00 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:57.767290 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d4fae86-771d-42c3-a709-1eec990dc33f None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:57.771567 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc08eb1-03d5-4e2e-92fe-e0b9f4146459 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:57.772321 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc08eb1-03d5-4e2e-92fe-e0b9f4146459 service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:57.772909 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc08eb1-03d5-4e2e-92fe-e0b9f4146459 service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:57.773252 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc08eb1-03d5-4e2e-92fe-e0b9f4146459 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:57.787183 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bc08eb1-03d5-4e2e-92fe-e0b9f4146459 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:57.788991 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bc08eb1-03d5-4e2e-92fe-e0b9f4146459 service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:57.825214 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1027/2055] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:53:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 133 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 11 00:53:57.835397 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-3fdd97b6-d458-48c9-b7bb-06569cc6bf00 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a41b64c71668411e90e980b92edb8a90'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:57.836246 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-3fdd97b6-d458-48c9-b7bb-06569cc6bf00 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a41b64c71668411e90e980b92edb8a90'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:57.836887 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-3fdd97b6-d458-48c9-b7bb-06569cc6bf00 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a41b64c71668411e90e980b92edb8a90'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:57.848715 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-3fdd97b6-d458-48c9-b7bb-06569cc6bf00 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a41b64c71668411e90e980b92edb8a90'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:57.861363 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:DELETE=14,keystone:SELECT=85,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:57.862774 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-3fdd97b6-d458-48c9-b7bb-06569cc6bf00 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a41b64c71668411e90e980b92edb8a90'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:57.891679 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1029/2056] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:53:57 2022] DELETE /identity/v3/users/a41b64c71668411e90e980b92edb8a90 => generated 0 bytes in 408 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:58.188454 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc08eb1-03d5-4e2e-92fe-e0b9f4146459 service neutron] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:58.193212 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-6bc08eb1-03d5-4e2e-92fe-e0b9f4146459 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:58.218236 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc08eb1-03d5-4e2e-92fe-e0b9f4146459 service neutron] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:58.337911 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1108ade6-1286-4711-85f0-1b6b77906494 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:58.338579 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1108ade6-1286-4711-85f0-1b6b77906494 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:58.338884 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1108ade6-1286-4711-85f0-1b6b77906494 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:58.339155 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1108ade6-1286-4711-85f0-1b6b77906494 admin admin] PATH_INFO: `/v3/projects/b6155e3b936b4675bfa9315475f66342` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:58.342457 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1108ade6-1286-4711-85f0-1b6b77906494 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b6155e3b936b4675bfa9315475f66342)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:58.344405 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1108ade6-1286-4711-85f0-1b6b77906494 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:58.394779 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:INSERT=1,keystone:DELETE=4,keystone:SELECT=95 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:58.407449 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-1108ade6-1286-4711-85f0-1b6b77906494 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b6155e3b936b4675bfa9315475f66342', 'request_id': 'req-1108ade6-1286-4711-85f0-1b6b77906494'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:58.413876 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-1108ade6-1286-4711-85f0-1b6b77906494 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b6155e3b936b4675bfa9315475f66342', 'request_id': 'req-1108ade6-1286-4711-85f0-1b6b77906494'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:58.423462 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-1108ade6-1286-4711-85f0-1b6b77906494 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b6155e3b936b4675bfa9315475f66342', 'request_id': 'req-1108ade6-1286-4711-85f0-1b6b77906494'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:58.439259 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-1108ade6-1286-4711-85f0-1b6b77906494 admin admin] The token cache is being invalidate because project b6155e3b936b4675bfa9315475f66342 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87924) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:53:58.439556 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-1108ade6-1286-4711-85f0-1b6b77906494 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:58.440222 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-1108ade6-1286-4711-85f0-1b6b77906494 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:58.441881 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1028/2057] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:53:58 2022] DELETE /identity/v3/projects/b6155e3b936b4675bfa9315475f66342 => generated 0 bytes in 255 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:58.446599 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f5ac91f-41b8-4cf8-a935-227e02d84b6d None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:58.446919 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f5ac91f-41b8-4cf8-a935-227e02d84b6d None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:58.447196 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f5ac91f-41b8-4cf8-a935-227e02d84b6d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:58.470032 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-5f5ac91f-41b8-4cf8-a935-227e02d84b6d None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:58.471377 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-5f5ac91f-41b8-4cf8-a935-227e02d84b6d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:58.481362 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1108ade6-1286-4711-85f0-1b6b77906494 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:58.482892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-1108ade6-1286-4711-85f0-1b6b77906494 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:58.548075 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1030/2058] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:53:58.554718 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1108ade6-1286-4711-85f0-1b6b77906494 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:58.556022 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa09c37-2a66-4b9b-991a-a21c1bdfd7e3 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:58.556403 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa09c37-2a66-4b9b-991a-a21c1bdfd7e3 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:58.556690 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa09c37-2a66-4b9b-991a-a21c1bdfd7e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:58.559950 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7f17d7-28ec-4cb4-9b6b-0cf8c5e36ffb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:58.560577 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7f17d7-28ec-4cb4-9b6b-0cf8c5e36ffb admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:58.560852 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7f17d7-28ec-4cb4-9b6b-0cf8c5e36ffb admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:58.561117 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7f17d7-28ec-4cb4-9b6b-0cf8c5e36ffb admin admin] PATH_INFO: `/v3/projects/383dadd676cb4a7cbdfbf40edce8df3e` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:58.562208 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b7f17d7-28ec-4cb4-9b6b-0cf8c5e36ffb admin admin] RBAC: Authorizing `identity:delete_project(project_id=383dadd676cb4a7cbdfbf40edce8df3e)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:58.564321 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b7f17d7-28ec-4cb4-9b6b-0cf8c5e36ffb admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:58.576746 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-bfa09c37-2a66-4b9b-991a-a21c1bdfd7e3 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:58.577998 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-bfa09c37-2a66-4b9b-991a-a21c1bdfd7e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:58.585757 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1031/2059] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:53:58.590520 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa09c37-2a66-4b9b-991a-a21c1bdfd7e3 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:58.600689 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa09c37-2a66-4b9b-991a-a21c1bdfd7e3 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:58.612440 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-5b7f17d7-28ec-4cb4-9b6b-0cf8c5e36ffb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '383dadd676cb4a7cbdfbf40edce8df3e', 'request_id': 'req-5b7f17d7-28ec-4cb4-9b6b-0cf8c5e36ffb'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:58.618526 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-5b7f17d7-28ec-4cb4-9b6b-0cf8c5e36ffb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '383dadd676cb4a7cbdfbf40edce8df3e', 'request_id': 'req-5b7f17d7-28ec-4cb4-9b6b-0cf8c5e36ffb'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:58.625964 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-5b7f17d7-28ec-4cb4-9b6b-0cf8c5e36ffb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '383dadd676cb4a7cbdfbf40edce8df3e', 'request_id': 'req-5b7f17d7-28ec-4cb4-9b6b-0cf8c5e36ffb'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:58.637879 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-5b7f17d7-28ec-4cb4-9b6b-0cf8c5e36ffb admin admin] The token cache is being invalidate because project 383dadd676cb4a7cbdfbf40edce8df3e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87924) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:53:58.638167 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-5b7f17d7-28ec-4cb4-9b6b-0cf8c5e36ffb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:58.638838 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-5b7f17d7-28ec-4cb4-9b6b-0cf8c5e36ffb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:58.640457 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1029/2060] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:53:58 2022] DELETE /identity/v3/projects/383dadd676cb4a7cbdfbf40edce8df3e => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:58.645329 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a31b854-2c2f-4710-9549-d0905a5474e4 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:58.645543 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a31b854-2c2f-4710-9549-d0905a5474e4 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:58.645814 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a31b854-2c2f-4710-9549-d0905a5474e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:58.658939 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c139f379-9a2a-4029-9b7a-59249637078f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:58.659654 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c139f379-9a2a-4029-9b7a-59249637078f admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:58.660197 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c139f379-9a2a-4029-9b7a-59249637078f admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:58.660462 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c139f379-9a2a-4029-9b7a-59249637078f admin admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:58.661105 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c139f379-9a2a-4029-9b7a-59249637078f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:53:58.661399 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c139f379-9a2a-4029-9b7a-59249637078f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:58.663007 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c139f379-9a2a-4029-9b7a-59249637078f admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:58.666140 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-0a31b854-2c2f-4710-9549-d0905a5474e4 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:58.667378 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-0a31b854-2c2f-4710-9549-d0905a5474e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:58.675099 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1032/2061] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 00:53:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:53:58.676040 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1030/2062] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:53:58.679585 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c139f379-9a2a-4029-9b7a-59249637078f admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:58.681312 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-c139f379-9a2a-4029-9b7a-59249637078f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:58.684611 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01295942-5667-4b68-bea8-0154c1465f9b None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:58.685105 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01295942-5667-4b68-bea8-0154c1465f9b None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:58.685195 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01295942-5667-4b68-bea8-0154c1465f9b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:58.694490 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c139f379-9a2a-4029-9b7a-59249637078f admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:58.699429 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b28d02-38a9-4cf1-ac1d-6b0f92c167fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:58.700017 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b28d02-38a9-4cf1-ac1d-6b0f92c167fc admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:58.700296 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b28d02-38a9-4cf1-ac1d-6b0f92c167fc admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:58.700574 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b28d02-38a9-4cf1-ac1d-6b0f92c167fc admin admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:58.701215 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b28d02-38a9-4cf1-ac1d-6b0f92c167fc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:58.703184 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b28d02-38a9-4cf1-ac1d-6b0f92c167fc admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:58.704231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-01295942-5667-4b68-bea8-0154c1465f9b None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:58.705404 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-01295942-5667-4b68-bea8-0154c1465f9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:58.713279 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1031/2063] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:53:58.718010 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01295942-5667-4b68-bea8-0154c1465f9b None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:58.727665 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01295942-5667-4b68-bea8-0154c1465f9b None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:58.732144 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1033/2064] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:53:58 2022] POST /identity/v3/projects => generated 369 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:53:58.733967 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-270adb8d-d097-4bd2-a659-92d55bfcb398 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:58.734592 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270adb8d-d097-4bd2-a659-92d55bfcb398 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:58.734896 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270adb8d-d097-4bd2-a659-92d55bfcb398 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:58.735253 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270adb8d-d097-4bd2-a659-92d55bfcb398 admin admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:58.735908 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-270adb8d-d097-4bd2-a659-92d55bfcb398 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:53:58.736203 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b28d02-38a9-4cf1-ac1d-6b0f92c167fc admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:58.736310 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-270adb8d-d097-4bd2-a659-92d55bfcb398 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:58.737802 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-270adb8d-d097-4bd2-a659-92d55bfcb398 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:58.750846 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1032/2065] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 00:53:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:53:58.754806 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-270adb8d-d097-4bd2-a659-92d55bfcb398 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:58.765498 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-270adb8d-d097-4bd2-a659-92d55bfcb398 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:58.794236 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=87,keystone:DELETE=11,keystone:INSERT=2 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:58.803374 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=97,keystone:INSERT=2,keystone:DELETE=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:58.816827 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b28d02-38a9-4cf1-ac1d-6b0f92c167fc admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:58.826483 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477fbdd6-4ba5-4b18-b61b-c0cff69017a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:58.827437 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477fbdd6-4ba5-4b18-b61b-c0cff69017a9 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:58.827699 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477fbdd6-4ba5-4b18-b61b-c0cff69017a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:58.828135 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477fbdd6-4ba5-4b18-b61b-c0cff69017a9 admin admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:58.828716 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-477fbdd6-4ba5-4b18-b61b-c0cff69017a9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:58.831026 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-477fbdd6-4ba5-4b18-b61b-c0cff69017a9 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:58.836900 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-441f8236-6752-49fd-813a-3eca3b74526a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:58.837874 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-441f8236-6752-49fd-813a-3eca3b74526a admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:58.838435 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-441f8236-6752-49fd-813a-3eca3b74526a admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:58.838942 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-441f8236-6752-49fd-813a-3eca3b74526a admin admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:58.840608 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-441f8236-6752-49fd-813a-3eca3b74526a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:58.847973 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-441f8236-6752-49fd-813a-3eca3b74526a admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:58.862230 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1034/2066] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:53:58 2022] POST /identity/v3/users => generated 285 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:53:58.863999 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477fbdd6-4ba5-4b18-b61b-c0cff69017a9 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:58.877500 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477fbdd6-4ba5-4b18-b61b-c0cff69017a9 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:58.884984 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcea50c2-2e8c-459a-9182-782f580f50ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:58.885856 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcea50c2-2e8c-459a-9182-782f580f50ea admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:58.887225 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcea50c2-2e8c-459a-9182-782f580f50ea admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:58.887939 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcea50c2-2e8c-459a-9182-782f580f50ea admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:58.888924 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcea50c2-2e8c-459a-9182-782f580f50ea admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:58.892092 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcea50c2-2e8c-459a-9182-782f580f50ea admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:58.907244 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1035/2067] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:53:58 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:53:58.911054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcea50c2-2e8c-459a-9182-782f580f50ea admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:58.917502 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1033/2068] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:53:58 2022] POST /identity/v3/projects => generated 361 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:53:58.921523 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-441f8236-6752-49fd-813a-3eca3b74526a admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:58.925504 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcea50c2-2e8c-459a-9182-782f580f50ea admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:59.020648 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa270f66-84d5-4c40-af19-fd7f38459231 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:59.022036 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa270f66-84d5-4c40-af19-fd7f38459231 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:59.022335 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa270f66-84d5-4c40-af19-fd7f38459231 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:59.023122 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa270f66-84d5-4c40-af19-fd7f38459231 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:59.024061 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa270f66-84d5-4c40-af19-fd7f38459231 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:59.026253 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa270f66-84d5-4c40-af19-fd7f38459231 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:59.033628 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-441f8236-6752-49fd-813a-3eca3b74526a admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:59.040038 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e544025-3081-45ed-906a-83fce99ee435 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:59.041029 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e544025-3081-45ed-906a-83fce99ee435 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:59.041427 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e544025-3081-45ed-906a-83fce99ee435 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:59.042204 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e544025-3081-45ed-906a-83fce99ee435 admin admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:59.042314 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1036/2069] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:53:58 2022] GET /identity/v3/roles => generated 1420 bytes in 131 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:53:59.045812 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e544025-3081-45ed-906a-83fce99ee435 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:59.046779 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa270f66-84d5-4c40-af19-fd7f38459231 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:59.048815 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e544025-3081-45ed-906a-83fce99ee435 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:59.062232 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa270f66-84d5-4c40-af19-fd7f38459231 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:59.068105 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a6006a-a295-49ab-b417-1acb6f9ed048 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:59.068852 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a6006a-a295-49ab-b417-1acb6f9ed048 admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:59.069204 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a6006a-a295-49ab-b417-1acb6f9ed048 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:59.069551 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a6006a-a295-49ab-b417-1acb6f9ed048 admin admin] PATH_INFO: `/v3/projects/59b634c104e744988203c197d7229e9a/users/b95b1812f54d49a5a552b6b569d326ee/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:59.079015 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1034/2070] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:53:58 2022] POST /identity/v3/users => generated 281 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:53:59.081289 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e544025-3081-45ed-906a-83fce99ee435 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:59.093477 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11a6006a-a295-49ab-b417-1acb6f9ed048 admin admin] RBAC: Authorizing `identity:create_grant(project_id=59b634c104e744988203c197d7229e9a, user_id=b95b1812f54d49a5a552b6b569d326ee, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:59.094908 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e544025-3081-45ed-906a-83fce99ee435 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:59.096460 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11a6006a-a295-49ab-b417-1acb6f9ed048 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:59.102020 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e66707-6297-40fd-a695-0382f578d062 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:59.102859 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e66707-6297-40fd-a695-0382f578d062 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:59.103250 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e66707-6297-40fd-a695-0382f578d062 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:59.103522 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e66707-6297-40fd-a695-0382f578d062 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:59.104375 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e66707-6297-40fd-a695-0382f578d062 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:59.107892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e66707-6297-40fd-a695-0382f578d062 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:59.118825 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1037/2071] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:53:59 2022] PUT /identity/v3/projects/59b634c104e744988203c197d7229e9a/users/b95b1812f54d49a5a552b6b569d326ee/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 73 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:53:59.128147 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1035/2072] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:53:59 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:53:59.131862 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a6006a-a295-49ab-b417-1acb6f9ed048 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:59.132607 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143ff409-1111-4419-bea4-05f5d8cec638 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:59.132909 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143ff409-1111-4419-bea4-05f5d8cec638 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:59.133157 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143ff409-1111-4419-bea4-05f5d8cec638 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:59.163350 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-143ff409-1111-4419-bea4-05f5d8cec638 None None] MFA Rules not processed for user `b95b1812f54d49a5a552b6b569d326ee`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:59.164808 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-143ff409-1111-4419-bea4-05f5d8cec638 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:59.194587 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a6006a-a295-49ab-b417-1acb6f9ed048 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:59.200265 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a41ac748-14c2-42e6-823c-0601b67d49a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:59.200867 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a41ac748-14c2-42e6-823c-0601b67d49a1 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:59.201143 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a41ac748-14c2-42e6-823c-0601b67d49a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:59.201412 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a41ac748-14c2-42e6-823c-0601b67d49a1 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:59.202089 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a41ac748-14c2-42e6-823c-0601b67d49a1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:59.204396 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a41ac748-14c2-42e6-823c-0601b67d49a1 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:59.218027 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1038/2073] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:53:59 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 11 00:53:59.219110 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a41ac748-14c2-42e6-823c-0601b67d49a1 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:59.228357 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a41ac748-14c2-42e6-823c-0601b67d49a1 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:59.230898 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1036/2074] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:59 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:53:59.233090 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6a39e0e-5cba-40ad-8dc8-3910d7febea4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:59.233728 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a39e0e-5cba-40ad-8dc8-3910d7febea4 admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:59.234002 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a39e0e-5cba-40ad-8dc8-3910d7febea4 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:59.234274 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a39e0e-5cba-40ad-8dc8-3910d7febea4 admin admin] PATH_INFO: `/v3/projects/88ccc98a4c864166a797348ebedef4be/users/715b2884b32d4e9ab1db494305aa17c1/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:59.242843 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143ff409-1111-4419-bea4-05f5d8cec638 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:59.244237 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-143ff409-1111-4419-bea4-05f5d8cec638 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:59.250140 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6a39e0e-5cba-40ad-8dc8-3910d7febea4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=88ccc98a4c864166a797348ebedef4be, user_id=715b2884b32d4e9ab1db494305aa17c1, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:59.252874 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6a39e0e-5cba-40ad-8dc8-3910d7febea4 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:59.266409 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1039/2075] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:53:59 2022] PUT /identity/v3/projects/88ccc98a4c864166a797348ebedef4be/users/715b2884b32d4e9ab1db494305aa17c1/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:53:59.273564 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:59.278840 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226ac587-a445-44cc-9e10-75b91f8eda7e None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:59.279347 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226ac587-a445-44cc-9e10-75b91f8eda7e None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:59.279729 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226ac587-a445-44cc-9e10-75b91f8eda7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:59.307547 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143ff409-1111-4419-bea4-05f5d8cec638 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:59.309452 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-226ac587-a445-44cc-9e10-75b91f8eda7e None None] MFA Rules not processed for user `715b2884b32d4e9ab1db494305aa17c1`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:59.314004 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-226ac587-a445-44cc-9e10-75b91f8eda7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:59.315907 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:59.369939 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1427516-72ff-46f0-b68e-52e3e327f901 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '06b1c911731f4267a5f37e91d948e7e9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:59.371186 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1427516-72ff-46f0-b68e-52e3e327f901 service glance] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:59.371709 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1427516-72ff-46f0-b68e-52e3e327f901 service glance] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:59.372112 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1427516-72ff-46f0-b68e-52e3e327f901 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:59.397634 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1040/2076] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:59 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:53:59.435767 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226ac587-a445-44cc-9e10-75b91f8eda7e None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:59.437430 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-226ac587-a445-44cc-9e10-75b91f8eda7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:59.457576 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1427516-72ff-46f0-b68e-52e3e327f901 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:59.459202 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1427516-72ff-46f0-b68e-52e3e327f901 service glance] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:59.468758 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1037/2077] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:53:59 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 226 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 11 00:53:59.499979 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226ac587-a445-44cc-9e10-75b91f8eda7e None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:59.503729 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33e6e37-f69d-4dfc-8a78-2c6be95c9ca2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:59.504327 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33e6e37-f69d-4dfc-8a78-2c6be95c9ca2 service cinder] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:59.504601 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33e6e37-f69d-4dfc-8a78-2c6be95c9ca2 service cinder] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:59.504868 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33e6e37-f69d-4dfc-8a78-2c6be95c9ca2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:59.513351 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b33e6e37-f69d-4dfc-8a78-2c6be95c9ca2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:59.514777 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b33e6e37-f69d-4dfc-8a78-2c6be95c9ca2 service cinder] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:59.524324 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1041/2078] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:53:59 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 11 00:53:59.541066 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1427516-72ff-46f0-b68e-52e3e327f901 service glance] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:59.609969 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1427516-72ff-46f0-b68e-52e3e327f901 service glance] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:59.614549 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b263222-e2a5-4e96-9576-4f0cf9b2165e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:59.615296 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b263222-e2a5-4e96-9576-4f0cf9b2165e admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:59.615569 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b263222-e2a5-4e96-9576-4f0cf9b2165e admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:59.615836 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b263222-e2a5-4e96-9576-4f0cf9b2165e admin admin] PATH_INFO: `/v3/users/715b2884b32d4e9ab1db494305aa17c1` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:59.616994 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b263222-e2a5-4e96-9576-4f0cf9b2165e admin admin] RBAC: Authorizing `identity:delete_user(user_id=715b2884b32d4e9ab1db494305aa17c1)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:59.618911 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b263222-e2a5-4e96-9576-4f0cf9b2165e admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:59.660832 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:53:59.673511 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-3b263222-e2a5-4e96-9576-4f0cf9b2165e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '715b2884b32d4e9ab1db494305aa17c1'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:59.674706 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-3b263222-e2a5-4e96-9576-4f0cf9b2165e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '715b2884b32d4e9ab1db494305aa17c1'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:59.675348 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-3b263222-e2a5-4e96-9576-4f0cf9b2165e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '715b2884b32d4e9ab1db494305aa17c1'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:59.683010 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-3b263222-e2a5-4e96-9576-4f0cf9b2165e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '715b2884b32d4e9ab1db494305aa17c1'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:59.697674 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-3b263222-e2a5-4e96-9576-4f0cf9b2165e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '715b2884b32d4e9ab1db494305aa17c1'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:53:59.712806 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1038/2079] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:53:59 2022] DELETE /identity/v3/users/715b2884b32d4e9ab1db494305aa17c1 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:53:59.718471 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1477cafb-9d82-45bd-88f5-e78cd16dca1b None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:59.718962 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1477cafb-9d82-45bd-88f5-e78cd16dca1b None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:59.719304 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1477cafb-9d82-45bd-88f5-e78cd16dca1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:59.740139 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-1477cafb-9d82-45bd-88f5-e78cd16dca1b None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:53:59.742015 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-1477cafb-9d82-45bd-88f5-e78cd16dca1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:59.809819 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1042/2080] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:53:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:53:59.820865 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b263222-e2a5-4e96-9576-4f0cf9b2165e admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:53:59.822276 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-3b263222-e2a5-4e96-9576-4f0cf9b2165e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:53:59.888137 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b263222-e2a5-4e96-9576-4f0cf9b2165e admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:53:59.888137 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f718bf-a496-43ea-b6fb-bc93c534969c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:53:59.890884 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f718bf-a496-43ea-b6fb-bc93c534969c service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:53:59.891245 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f718bf-a496-43ea-b6fb-bc93c534969c service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:53:59.891532 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f718bf-a496-43ea-b6fb-bc93c534969c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:53:59.901743 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82f718bf-a496-43ea-b6fb-bc93c534969c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:53:59.903238 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82f718bf-a496-43ea-b6fb-bc93c534969c service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:53:59.914859 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1039/2081] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:53:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 11 00:54:00.232092 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1477cafb-9d82-45bd-88f5-e78cd16dca1b None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:00.233822 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-1477cafb-9d82-45bd-88f5-e78cd16dca1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:00.249784 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1477cafb-9d82-45bd-88f5-e78cd16dca1b None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:00.263474 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b8750f5-d118-4f39-a645-1d63079cac31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:00.264125 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8750f5-d118-4f39-a645-1d63079cac31 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:00.265310 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8750f5-d118-4f39-a645-1d63079cac31 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:00.265310 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8750f5-d118-4f39-a645-1d63079cac31 admin admin] PATH_INFO: `/v3/projects/88ccc98a4c864166a797348ebedef4be` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:00.266399 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b8750f5-d118-4f39-a645-1d63079cac31 admin admin] RBAC: Authorizing `identity:delete_project(project_id=88ccc98a4c864166a797348ebedef4be)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:00.268362 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b8750f5-d118-4f39-a645-1d63079cac31 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:00.323124 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:00.330742 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-6b8750f5-d118-4f39-a645-1d63079cac31 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '88ccc98a4c864166a797348ebedef4be', 'request_id': 'req-6b8750f5-d118-4f39-a645-1d63079cac31'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:00.337421 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-6b8750f5-d118-4f39-a645-1d63079cac31 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '88ccc98a4c864166a797348ebedef4be', 'request_id': 'req-6b8750f5-d118-4f39-a645-1d63079cac31'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:00.346663 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-6b8750f5-d118-4f39-a645-1d63079cac31 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '88ccc98a4c864166a797348ebedef4be', 'request_id': 'req-6b8750f5-d118-4f39-a645-1d63079cac31'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:00.363983 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-6b8750f5-d118-4f39-a645-1d63079cac31 admin admin] The token cache is being invalidate because project 88ccc98a4c864166a797348ebedef4be was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87923) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:54:00.364280 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-6b8750f5-d118-4f39-a645-1d63079cac31 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:00.365025 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-6b8750f5-d118-4f39-a645-1d63079cac31 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:00.366586 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1043/2082] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:54:00 2022] DELETE /identity/v3/projects/88ccc98a4c864166a797348ebedef4be => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:54:00.371670 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ce2136-bc24-4e43-a847-4b94b9576224 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:00.372216 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ce2136-bc24-4e43-a847-4b94b9576224 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:00.372715 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ce2136-bc24-4e43-a847-4b94b9576224 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:00.409744 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-e2ce2136-bc24-4e43-a847-4b94b9576224 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:00.411316 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-e2ce2136-bc24-4e43-a847-4b94b9576224 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:00.471824 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b8750f5-d118-4f39-a645-1d63079cac31 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:00.473462 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-6b8750f5-d118-4f39-a645-1d63079cac31 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:00.482616 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1040/2083] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:54:00.489961 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6512fcb3-7baf-430b-9aa5-25d097620d3b None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:00.490245 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6512fcb3-7baf-430b-9aa5-25d097620d3b None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:00.490515 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6512fcb3-7baf-430b-9aa5-25d097620d3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:00.511961 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-6512fcb3-7baf-430b-9aa5-25d097620d3b None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:00.513378 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-6512fcb3-7baf-430b-9aa5-25d097620d3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:00.522115 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1041/2084] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:54:00.526198 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6512fcb3-7baf-430b-9aa5-25d097620d3b None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:00.536299 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6512fcb3-7baf-430b-9aa5-25d097620d3b None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:00.539185 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b8750f5-d118-4f39-a645-1d63079cac31 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:00.541664 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5636b9da-377f-4ed6-aa77-1d6a02885f85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:00.542469 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5636b9da-377f-4ed6-aa77-1d6a02885f85 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:00.543078 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5636b9da-377f-4ed6-aa77-1d6a02885f85 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:00.543515 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5636b9da-377f-4ed6-aa77-1d6a02885f85 admin admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:00.544403 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5636b9da-377f-4ed6-aa77-1d6a02885f85 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:54:00.544630 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eaf5462-ebd4-42e3-b1b6-5807fc9eee6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:00.544974 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5636b9da-377f-4ed6-aa77-1d6a02885f85 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:00.545269 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eaf5462-ebd4-42e3-b1b6-5807fc9eee6d admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:00.545539 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eaf5462-ebd4-42e3-b1b6-5807fc9eee6d admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:00.545877 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eaf5462-ebd4-42e3-b1b6-5807fc9eee6d admin admin] PATH_INFO: `/v3/users/2f87d0a253674c86b093ffcb590e07b1` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:00.546986 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eaf5462-ebd4-42e3-b1b6-5807fc9eee6d admin admin] RBAC: Authorizing `identity:delete_user(user_id=2f87d0a253674c86b093ffcb590e07b1)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:00.547263 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5636b9da-377f-4ed6-aa77-1d6a02885f85 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:00.548897 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eaf5462-ebd4-42e3-b1b6-5807fc9eee6d admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:00.559460 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1042/2085] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 00:54:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:54:00.563718 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5636b9da-377f-4ed6-aa77-1d6a02885f85 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:00.573369 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5636b9da-377f-4ed6-aa77-1d6a02885f85 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:00.578889 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ff8e53-cb14-416c-87a2-f6e56efb70f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:00.579482 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ff8e53-cb14-416c-87a2-f6e56efb70f9 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:00.579757 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ff8e53-cb14-416c-87a2-f6e56efb70f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:00.580057 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ff8e53-cb14-416c-87a2-f6e56efb70f9 admin admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:00.580681 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00ff8e53-cb14-416c-87a2-f6e56efb70f9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:00.582570 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00ff8e53-cb14-416c-87a2-f6e56efb70f9 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:00.601041 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-5eaf5462-ebd4-42e3-b1b6-5807fc9eee6d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2f87d0a253674c86b093ffcb590e07b1'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:00.601707 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-5eaf5462-ebd4-42e3-b1b6-5807fc9eee6d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2f87d0a253674c86b093ffcb590e07b1'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:00.602335 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-5eaf5462-ebd4-42e3-b1b6-5807fc9eee6d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2f87d0a253674c86b093ffcb590e07b1'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:00.611892 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-5eaf5462-ebd4-42e3-b1b6-5807fc9eee6d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2f87d0a253674c86b093ffcb590e07b1'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:00.612154 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1043/2086] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:54:00 2022] POST /identity/v3/projects => generated 377 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:54:00.616007 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ff8e53-cb14-416c-87a2-f6e56efb70f9 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:00.617651 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-00ff8e53-cb14-416c-87a2-f6e56efb70f9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:00.627189 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-5eaf5462-ebd4-42e3-b1b6-5807fc9eee6d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2f87d0a253674c86b093ffcb590e07b1'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:00.632176 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=93,keystone:DELETE=5,keystone:INSERT=2 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:00.641046 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1044/2087] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:54:00 2022] DELETE /identity/v3/users/2f87d0a253674c86b093ffcb590e07b1 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:54:00.645359 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2124adc9-2722-4504-bce3-c9aadc507486 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:00.645644 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2124adc9-2722-4504-bce3-c9aadc507486 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:00.645914 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2124adc9-2722-4504-bce3-c9aadc507486 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:00.665498 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-2124adc9-2722-4504-bce3-c9aadc507486 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:00.666690 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-2124adc9-2722-4504-bce3-c9aadc507486 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:00.689343 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ff8e53-cb14-416c-87a2-f6e56efb70f9 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:00.717927 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:DELETE=14,keystone:SELECT=84,keystone:INSERT=2 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:00.733489 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1045/2088] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:54:00.743700 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2124adc9-2722-4504-bce3-c9aadc507486 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:00.746154 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-2124adc9-2722-4504-bce3-c9aadc507486 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:00.753275 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d884257-08c9-4486-987b-33dab8af8ae5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:00.753887 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d884257-08c9-4486-987b-33dab8af8ae5 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:00.754162 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d884257-08c9-4486-987b-33dab8af8ae5 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:00.754430 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d884257-08c9-4486-987b-33dab8af8ae5 admin admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:00.755093 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d884257-08c9-4486-987b-33dab8af8ae5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:00.757024 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d884257-08c9-4486-987b-33dab8af8ae5 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:00.779130 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1044/2089] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:54:00 2022] POST /identity/v3/users => generated 289 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:54:00.783812 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d884257-08c9-4486-987b-33dab8af8ae5 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:00.796867 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d884257-08c9-4486-987b-33dab8af8ae5 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:00.802278 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3174741a-284f-4ba4-9b5b-a4adc6afa379 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:00.802920 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3174741a-284f-4ba4-9b5b-a4adc6afa379 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:00.803202 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3174741a-284f-4ba4-9b5b-a4adc6afa379 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:00.803472 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3174741a-284f-4ba4-9b5b-a4adc6afa379 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:00.804118 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3174741a-284f-4ba4-9b5b-a4adc6afa379 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:00.805748 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3174741a-284f-4ba4-9b5b-a4adc6afa379 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:00.812590 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2124adc9-2722-4504-bce3-c9aadc507486 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:00.816371 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84df3693-a99e-430d-b03b-9a4baa310f5a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:00.816974 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84df3693-a99e-430d-b03b-9a4baa310f5a service neutron] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:00.817251 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84df3693-a99e-430d-b03b-9a4baa310f5a service neutron] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:00.817443 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1045/2090] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:54:00 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 11 00:54:00.820690 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84df3693-a99e-430d-b03b-9a4baa310f5a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:00.822000 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3174741a-284f-4ba4-9b5b-a4adc6afa379 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:00.827534 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84df3693-a99e-430d-b03b-9a4baa310f5a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:00.828988 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84df3693-a99e-430d-b03b-9a4baa310f5a service neutron] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:00.833631 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3174741a-284f-4ba4-9b5b-a4adc6afa379 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:00.838952 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-757450d9-9dda-4264-a2a5-3cfc84146471 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:00.840094 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1046/2091] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:54:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 11 00:54:00.840416 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757450d9-9dda-4264-a2a5-3cfc84146471 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:00.840654 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757450d9-9dda-4264-a2a5-3cfc84146471 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:00.840881 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757450d9-9dda-4264-a2a5-3cfc84146471 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:00.841567 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-757450d9-9dda-4264-a2a5-3cfc84146471 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:00.843580 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-757450d9-9dda-4264-a2a5-3cfc84146471 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:00.860466 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1046/2092] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:54:00 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:54:00.861638 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84df3693-a99e-430d-b03b-9a4baa310f5a service neutron] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:00.872165 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84df3693-a99e-430d-b03b-9a4baa310f5a service neutron] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:00.878722 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ca8e9a-7575-4f4d-ac48-45f5d9b18b04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:00.879844 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ca8e9a-7575-4f4d-ac48-45f5d9b18b04 admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:00.880122 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ca8e9a-7575-4f4d-ac48-45f5d9b18b04 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:00.880395 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ca8e9a-7575-4f4d-ac48-45f5d9b18b04 admin admin] PATH_INFO: `/v3/projects/bc794a63b6db4ba3b2f7293767c998fa/users/ea94872269d14d05be54048753f683a1/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:00.896959 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49ca8e9a-7575-4f4d-ac48-45f5d9b18b04 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bc794a63b6db4ba3b2f7293767c998fa, user_id=ea94872269d14d05be54048753f683a1, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:00.900229 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49ca8e9a-7575-4f4d-ac48-45f5d9b18b04 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:00.915357 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1047/2093] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:54:00 2022] PUT /identity/v3/projects/bc794a63b6db4ba3b2f7293767c998fa/users/ea94872269d14d05be54048753f683a1/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:54:00.927282 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da5b84a-996d-4fca-a705-58d680a5d922 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:00.927731 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da5b84a-996d-4fca-a705-58d680a5d922 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:00.928102 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da5b84a-996d-4fca-a705-58d680a5d922 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:00.962234 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-0da5b84a-996d-4fca-a705-58d680a5d922 None None] MFA Rules not processed for user `ea94872269d14d05be54048753f683a1`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:00.963682 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-0da5b84a-996d-4fca-a705-58d680a5d922 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:01.049441 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1047/2094] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:00 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:54:01.061718 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ca8e9a-7575-4f4d-ac48-45f5d9b18b04 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:01.063631 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-49ca8e9a-7575-4f4d-ac48-45f5d9b18b04 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:01.130078 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ca8e9a-7575-4f4d-ac48-45f5d9b18b04 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:01.133755 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49c63cb-670c-40c0-b36e-0850ddc9411f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:01.134363 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49c63cb-670c-40c0-b36e-0850ddc9411f service cinder] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:01.134648 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49c63cb-670c-40c0-b36e-0850ddc9411f service cinder] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:01.134950 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49c63cb-670c-40c0-b36e-0850ddc9411f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:01.144783 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da5b84a-996d-4fca-a705-58d680a5d922 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:01.146388 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-0da5b84a-996d-4fca-a705-58d680a5d922 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:01.148085 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f49c63cb-670c-40c0-b36e-0850ddc9411f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:01.149598 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f49c63cb-670c-40c0-b36e-0850ddc9411f service cinder] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:01.170370 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1048/2095] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:54:01 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:54:01.173869 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:01.222728 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da5b84a-996d-4fca-a705-58d680a5d922 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:01.228686 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3a28fc0-780f-47eb-8f93-a176a83ef2ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:01.229642 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a28fc0-780f-47eb-8f93-a176a83ef2ce admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:01.230319 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a28fc0-780f-47eb-8f93-a176a83ef2ce admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:01.230599 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a28fc0-780f-47eb-8f93-a176a83ef2ce admin admin] PATH_INFO: `/v3/projects/12d8edb251ec4b20a3a383572f12fc52` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:01.231814 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3a28fc0-780f-47eb-8f93-a176a83ef2ce admin admin] RBAC: Authorizing `identity:delete_project(project_id=12d8edb251ec4b20a3a383572f12fc52)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:01.233755 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3a28fc0-780f-47eb-8f93-a176a83ef2ce admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:01.288050 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-e3a28fc0-780f-47eb-8f93-a176a83ef2ce admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '12d8edb251ec4b20a3a383572f12fc52', 'request_id': 'req-e3a28fc0-780f-47eb-8f93-a176a83ef2ce'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:01.294385 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-e3a28fc0-780f-47eb-8f93-a176a83ef2ce admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '12d8edb251ec4b20a3a383572f12fc52', 'request_id': 'req-e3a28fc0-780f-47eb-8f93-a176a83ef2ce'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:01.303429 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-e3a28fc0-780f-47eb-8f93-a176a83ef2ce admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '12d8edb251ec4b20a3a383572f12fc52', 'request_id': 'req-e3a28fc0-780f-47eb-8f93-a176a83ef2ce'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:01.314965 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-e3a28fc0-780f-47eb-8f93-a176a83ef2ce admin admin] The token cache is being invalidate because project 12d8edb251ec4b20a3a383572f12fc52 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87924) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:54:01.315251 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-e3a28fc0-780f-47eb-8f93-a176a83ef2ce admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:01.315895 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-e3a28fc0-780f-47eb-8f93-a176a83ef2ce admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:01.317626 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1048/2096] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:54:01 2022] DELETE /identity/v3/projects/12d8edb251ec4b20a3a383572f12fc52 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:54:01.321804 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49c63cb-670c-40c0-b36e-0850ddc9411f service cinder] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:01.334565 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49c63cb-670c-40c0-b36e-0850ddc9411f service cinder] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:01.417454 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d48af031-de73-4e93-aa3c-a48ceadd4d1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:01.418151 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48af031-de73-4e93-aa3c-a48ceadd4d1e admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:01.418679 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48af031-de73-4e93-aa3c-a48ceadd4d1e admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:01.419185 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48af031-de73-4e93-aa3c-a48ceadd4d1e admin admin] PATH_INFO: `/v3/users/15362a7e5cba4d6f86ded3d91654239f` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:01.420510 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d48af031-de73-4e93-aa3c-a48ceadd4d1e admin admin] RBAC: Authorizing `identity:delete_user(user_id=15362a7e5cba4d6f86ded3d91654239f)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:01.422917 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d48af031-de73-4e93-aa3c-a48ceadd4d1e admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:01.443436 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=98,keystone:INSERT=1,keystone:DELETE=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:01.495282 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-d48af031-de73-4e93-aa3c-a48ceadd4d1e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '15362a7e5cba4d6f86ded3d91654239f'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:01.497772 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-d48af031-de73-4e93-aa3c-a48ceadd4d1e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '15362a7e5cba4d6f86ded3d91654239f'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:01.503515 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-d48af031-de73-4e93-aa3c-a48ceadd4d1e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '15362a7e5cba4d6f86ded3d91654239f'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:01.527884 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-d48af031-de73-4e93-aa3c-a48ceadd4d1e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '15362a7e5cba4d6f86ded3d91654239f'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:01.565686 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-d48af031-de73-4e93-aa3c-a48ceadd4d1e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '15362a7e5cba4d6f86ded3d91654239f'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:01.596319 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1049/2097] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:54:01 2022] DELETE /identity/v3/users/15362a7e5cba4d6f86ded3d91654239f => generated 0 bytes in 275 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:54:02.099049 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3a28fc0-780f-47eb-8f93-a176a83ef2ce admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:02.100487 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-e3a28fc0-780f-47eb-8f93-a176a83ef2ce admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:02.113277 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3a28fc0-780f-47eb-8f93-a176a83ef2ce admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:02.173563 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1748bb74-7fc4-4316-953b-842fa1cd0a23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:02.174212 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1748bb74-7fc4-4316-953b-842fa1cd0a23 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:02.174428 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1748bb74-7fc4-4316-953b-842fa1cd0a23 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:02.174654 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1748bb74-7fc4-4316-953b-842fa1cd0a23 admin admin] PATH_INFO: `/v3/projects/d297158cd9e341e4a094266fef68fa07` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:02.175914 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1748bb74-7fc4-4316-953b-842fa1cd0a23 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d297158cd9e341e4a094266fef68fa07)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:02.177881 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1748bb74-7fc4-4316-953b-842fa1cd0a23 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:02.227424 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-1748bb74-7fc4-4316-953b-842fa1cd0a23 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd297158cd9e341e4a094266fef68fa07', 'request_id': 'req-1748bb74-7fc4-4316-953b-842fa1cd0a23'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:02.228486 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:02.236065 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-1748bb74-7fc4-4316-953b-842fa1cd0a23 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd297158cd9e341e4a094266fef68fa07', 'request_id': 'req-1748bb74-7fc4-4316-953b-842fa1cd0a23'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:02.244546 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-1748bb74-7fc4-4316-953b-842fa1cd0a23 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd297158cd9e341e4a094266fef68fa07', 'request_id': 'req-1748bb74-7fc4-4316-953b-842fa1cd0a23'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:02.256221 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-1748bb74-7fc4-4316-953b-842fa1cd0a23 admin admin] The token cache is being invalidate because project d297158cd9e341e4a094266fef68fa07 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87924) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:54:02.256639 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-1748bb74-7fc4-4316-953b-842fa1cd0a23 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:02.257368 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-1748bb74-7fc4-4316-953b-842fa1cd0a23 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:02.259077 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1049/2098] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:54:02 2022] DELETE /identity/v3/projects/d297158cd9e341e4a094266fef68fa07 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:54:02.263919 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a9e0e2b-776d-441b-9e41-ebde4f178e82 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:02.264279 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a9e0e2b-776d-441b-9e41-ebde4f178e82 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:02.264616 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a9e0e2b-776d-441b-9e41-ebde4f178e82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:02.285665 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-4a9e0e2b-776d-441b-9e41-ebde4f178e82 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:02.286992 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-4a9e0e2b-776d-441b-9e41-ebde4f178e82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:02.353116 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1050/2099] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:54:02.361324 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23ccb4c-2f1f-4c3c-9505-74af24ad003f None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:02.361638 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23ccb4c-2f1f-4c3c-9505-74af24ad003f None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:02.361930 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23ccb4c-2f1f-4c3c-9505-74af24ad003f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:02.384313 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-a23ccb4c-2f1f-4c3c-9505-74af24ad003f None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:02.385533 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-a23ccb4c-2f1f-4c3c-9505-74af24ad003f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:02.395261 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1050/2100] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:54:02.400681 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a9e0e2b-776d-441b-9e41-ebde4f178e82 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:02.411703 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a9e0e2b-776d-441b-9e41-ebde4f178e82 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:02.416924 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04b385c8-5f8c-4e61-ab98-f8cdd49a0e3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:02.417577 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b385c8-5f8c-4e61-ab98-f8cdd49a0e3b admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:02.417856 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b385c8-5f8c-4e61-ab98-f8cdd49a0e3b admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:02.418257 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b385c8-5f8c-4e61-ab98-f8cdd49a0e3b admin admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:02.418958 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04b385c8-5f8c-4e61-ab98-f8cdd49a0e3b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:54:02.419296 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04b385c8-5f8c-4e61-ab98-f8cdd49a0e3b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:02.421082 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04b385c8-5f8c-4e61-ab98-f8cdd49a0e3b admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:02.435623 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1051/2101] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 00:54:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:54:02.440244 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a23ccb4c-2f1f-4c3c-9505-74af24ad003f None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:02.450857 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a23ccb4c-2f1f-4c3c-9505-74af24ad003f None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:02.455966 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-851dbd91-d5fc-41ab-b10b-36beb1730939 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:02.456671 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851dbd91-d5fc-41ab-b10b-36beb1730939 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:02.457010 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851dbd91-d5fc-41ab-b10b-36beb1730939 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:02.457355 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851dbd91-d5fc-41ab-b10b-36beb1730939 admin admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:02.458065 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-851dbd91-d5fc-41ab-b10b-36beb1730939 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:02.460129 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-851dbd91-d5fc-41ab-b10b-36beb1730939 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:02.496287 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1051/2102] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:54:02 2022] POST /identity/v3/projects => generated 385 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:54:02.497807 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04b385c8-5f8c-4e61-ab98-f8cdd49a0e3b admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:02.508297 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04b385c8-5f8c-4e61-ab98-f8cdd49a0e3b admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:02.568443 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-014ef8fb-36b3-40af-988a-0eb7001e520e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:02.569014 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014ef8fb-36b3-40af-988a-0eb7001e520e admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:02.569662 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014ef8fb-36b3-40af-988a-0eb7001e520e admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:02.569662 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014ef8fb-36b3-40af-988a-0eb7001e520e admin admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:02.570215 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-014ef8fb-36b3-40af-988a-0eb7001e520e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:02.572401 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-014ef8fb-36b3-40af-988a-0eb7001e520e admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:02.600598 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1052/2103] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:54:02 2022] POST /identity/v3/users => generated 293 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:54:02.601479 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-851dbd91-d5fc-41ab-b10b-36beb1730939 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:02.643078 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-851dbd91-d5fc-41ab-b10b-36beb1730939 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:02.643398 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b36b5c-1189-4155-b9ae-09d8a8956678 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:02.644064 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b36b5c-1189-4155-b9ae-09d8a8956678 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:02.644477 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b36b5c-1189-4155-b9ae-09d8a8956678 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:02.644899 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b36b5c-1189-4155-b9ae-09d8a8956678 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:02.645745 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61b36b5c-1189-4155-b9ae-09d8a8956678 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:02.647536 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61b36b5c-1189-4155-b9ae-09d8a8956678 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:02.668624 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1052/2104] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:54:02 2022] GET /identity/v3/roles => generated 1420 bytes in 68 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:54:02.672905 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-014ef8fb-36b3-40af-988a-0eb7001e520e admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:02.686989 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-014ef8fb-36b3-40af-988a-0eb7001e520e admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:02.692436 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a02b5fe-0af2-4501-8297-9310c53c9fb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:02.693230 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a02b5fe-0af2-4501-8297-9310c53c9fb3 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:02.693567 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a02b5fe-0af2-4501-8297-9310c53c9fb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:02.694209 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a02b5fe-0af2-4501-8297-9310c53c9fb3 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:02.695169 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a02b5fe-0af2-4501-8297-9310c53c9fb3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:02.697316 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a02b5fe-0af2-4501-8297-9310c53c9fb3 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:02.706682 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:DELETE=9,keystone:SELECT=87,keystone:INSERT=4 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:02.722974 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1053/2105] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:54:02 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 11 00:54:02.724026 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b36b5c-1189-4155-b9ae-09d8a8956678 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:02.737003 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b36b5c-1189-4155-b9ae-09d8a8956678 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:02.744691 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194944ab-5aca-4969-bcea-9f07a38278c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:02.744691 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194944ab-5aca-4969-bcea-9f07a38278c2 admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:02.744691 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194944ab-5aca-4969-bcea-9f07a38278c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:02.745031 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194944ab-5aca-4969-bcea-9f07a38278c2 admin admin] PATH_INFO: `/v3/projects/ff59703649fa43269925aa9065689849/users/90b14790bd90497c8d6c5f6705e0d44d/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:02.783972 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194944ab-5aca-4969-bcea-9f07a38278c2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ff59703649fa43269925aa9065689849, user_id=90b14790bd90497c8d6c5f6705e0d44d, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:02.791712 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194944ab-5aca-4969-bcea-9f07a38278c2 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:02.821125 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1053/2106] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:54:02 2022] PUT /identity/v3/projects/ff59703649fa43269925aa9065689849/users/90b14790bd90497c8d6c5f6705e0d44d/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:54:02.831469 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50cc6990-d83c-4d56-9730-af08fdf51ce2 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:02.831818 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50cc6990-d83c-4d56-9730-af08fdf51ce2 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:02.832075 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50cc6990-d83c-4d56-9730-af08fdf51ce2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:02.900481 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-50cc6990-d83c-4d56-9730-af08fdf51ce2 None None] MFA Rules not processed for user `90b14790bd90497c8d6c5f6705e0d44d`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:02.900481 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-50cc6990-d83c-4d56-9730-af08fdf51ce2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:03.035888 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1054/2107] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:02 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:54:03.043605 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194944ab-5aca-4969-bcea-9f07a38278c2 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:03.144553 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194944ab-5aca-4969-bcea-9f07a38278c2 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:03.157169 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb0cf3d6-2f4a-4b1d-bfa3-499e694a2ab4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:03.157984 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb0cf3d6-2f4a-4b1d-bfa3-499e694a2ab4 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:03.158432 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb0cf3d6-2f4a-4b1d-bfa3-499e694a2ab4 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:03.158769 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb0cf3d6-2f4a-4b1d-bfa3-499e694a2ab4 admin admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:03.159524 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb0cf3d6-2f4a-4b1d-bfa3-499e694a2ab4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:03.161800 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb0cf3d6-2f4a-4b1d-bfa3-499e694a2ab4 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:03.195607 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1054/2108] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:54:03 2022] POST /identity/v3/projects => generated 387 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:54:03.199792 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50cc6990-d83c-4d56-9730-af08fdf51ce2 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:03.212619 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50cc6990-d83c-4d56-9730-af08fdf51ce2 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:03.280146 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8880e020-1986-4ac2-b971-61282fff344d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:03.280665 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8880e020-1986-4ac2-b971-61282fff344d admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:03.280946 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8880e020-1986-4ac2-b971-61282fff344d admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:03.281218 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8880e020-1986-4ac2-b971-61282fff344d admin admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:03.281854 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8880e020-1986-4ac2-b971-61282fff344d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:03.283835 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8880e020-1986-4ac2-b971-61282fff344d admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:03.308547 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1055/2109] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:54:03 2022] POST /identity/v3/users => generated 294 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:54:03.312898 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb0cf3d6-2f4a-4b1d-bfa3-499e694a2ab4 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:03.325457 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb0cf3d6-2f4a-4b1d-bfa3-499e694a2ab4 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:03.330742 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d3cc68-d07c-4607-ab45-1129c20f7d70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:03.331153 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d3cc68-d07c-4607-ab45-1129c20f7d70 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:03.331430 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d3cc68-d07c-4607-ab45-1129c20f7d70 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:03.331697 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d3cc68-d07c-4607-ab45-1129c20f7d70 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:03.332354 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8d3cc68-d07c-4607-ab45-1129c20f7d70 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:03.334145 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8d3cc68-d07c-4607-ab45-1129c20f7d70 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:03.349616 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1055/2110] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:54:03 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:54:03.353780 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8880e020-1986-4ac2-b971-61282fff344d admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:03.365901 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8880e020-1986-4ac2-b971-61282fff344d admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:03.370731 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78602800-0b22-4553-ab4f-9ffb452f0875 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:03.371372 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78602800-0b22-4553-ab4f-9ffb452f0875 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:03.371651 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78602800-0b22-4553-ab4f-9ffb452f0875 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:03.371926 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78602800-0b22-4553-ab4f-9ffb452f0875 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:03.372579 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78602800-0b22-4553-ab4f-9ffb452f0875 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:03.374236 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78602800-0b22-4553-ab4f-9ffb452f0875 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:03.385172 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1056/2111] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:54:03 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:54:03.390381 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d3cc68-d07c-4607-ab45-1129c20f7d70 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:03.400972 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d3cc68-d07c-4607-ab45-1129c20f7d70 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:03.411817 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad85afe-a6a5-4782-a7fc-0c3c4e68ef21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:03.412491 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad85afe-a6a5-4782-a7fc-0c3c4e68ef21 admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:03.412772 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad85afe-a6a5-4782-a7fc-0c3c4e68ef21 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:03.413045 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad85afe-a6a5-4782-a7fc-0c3c4e68ef21 admin admin] PATH_INFO: `/v3/projects/569a9686a0244e2781f1e9beca6bc74a/users/e5d00c46be13447395f6f09015de93d1/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:03.428707 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cad85afe-a6a5-4782-a7fc-0c3c4e68ef21 admin admin] RBAC: Authorizing `identity:create_grant(project_id=569a9686a0244e2781f1e9beca6bc74a, user_id=e5d00c46be13447395f6f09015de93d1, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:03.431515 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cad85afe-a6a5-4782-a7fc-0c3c4e68ef21 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:03.445027 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1056/2112] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:54:03 2022] PUT /identity/v3/projects/569a9686a0244e2781f1e9beca6bc74a/users/e5d00c46be13447395f6f09015de93d1/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:54:03.456722 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3adebea0-91d2-4343-9a32-d9aed776f541 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:03.457011 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3adebea0-91d2-4343-9a32-d9aed776f541 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:03.457290 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3adebea0-91d2-4343-9a32-d9aed776f541 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:03.487879 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-3adebea0-91d2-4343-9a32-d9aed776f541 None None] MFA Rules not processed for user `e5d00c46be13447395f6f09015de93d1`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:03.489089 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-3adebea0-91d2-4343-9a32-d9aed776f541 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:03.556156 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:03.562574 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1057/2113] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:03 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:54:03.567076 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad85afe-a6a5-4782-a7fc-0c3c4e68ef21 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:03.598913 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=92,keystone:INSERT=5,keystone:DELETE=3 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:03.643236 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad85afe-a6a5-4782-a7fc-0c3c4e68ef21 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:03.648377 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fed5955-cb1a-4d5c-8b05-f0b8a29e7a31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:03.649067 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fed5955-cb1a-4d5c-8b05-f0b8a29e7a31 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:03.649471 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fed5955-cb1a-4d5c-8b05-f0b8a29e7a31 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:03.649821 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fed5955-cb1a-4d5c-8b05-f0b8a29e7a31 admin admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:03.650524 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fed5955-cb1a-4d5c-8b05-f0b8a29e7a31 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:03.652585 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fed5955-cb1a-4d5c-8b05-f0b8a29e7a31 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:03.690326 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1057/2114] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:54:03 2022] POST /identity/v3/projects => generated 387 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:54:03.694646 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3adebea0-91d2-4343-9a32-d9aed776f541 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:03.715166 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3adebea0-91d2-4343-9a32-d9aed776f541 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:03.778058 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-607e5754-841d-4c0c-a8f6-b3f8810f3d7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:03.778761 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-607e5754-841d-4c0c-a8f6-b3f8810f3d7b admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:03.779071 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-607e5754-841d-4c0c-a8f6-b3f8810f3d7b admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:03.779420 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-607e5754-841d-4c0c-a8f6-b3f8810f3d7b admin admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:03.780164 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-607e5754-841d-4c0c-a8f6-b3f8810f3d7b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:03.782226 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-607e5754-841d-4c0c-a8f6-b3f8810f3d7b admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:03.814206 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1058/2115] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:54:03 2022] POST /identity/v3/users => generated 300 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:54:03.817889 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fed5955-cb1a-4d5c-8b05-f0b8a29e7a31 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:03.829534 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fed5955-cb1a-4d5c-8b05-f0b8a29e7a31 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:03.834180 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac7f6a7c-a4a0-433a-a985-e7df4a781691 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:03.834937 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7f6a7c-a4a0-433a-a985-e7df4a781691 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:03.835216 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7f6a7c-a4a0-433a-a985-e7df4a781691 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:03.835594 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7f6a7c-a4a0-433a-a985-e7df4a781691 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:03.836246 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac7f6a7c-a4a0-433a-a985-e7df4a781691 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:03.837966 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac7f6a7c-a4a0-433a-a985-e7df4a781691 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:03.849647 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1058/2116] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:54:03 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:54:03.853801 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-607e5754-841d-4c0c-a8f6-b3f8810f3d7b admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:03.866399 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-607e5754-841d-4c0c-a8f6-b3f8810f3d7b admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:03.872013 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1a1550-0af6-47e9-a1e7-eb874f124845 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:03.873098 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1a1550-0af6-47e9-a1e7-eb874f124845 admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:03.873098 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1a1550-0af6-47e9-a1e7-eb874f124845 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:03.873291 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1a1550-0af6-47e9-a1e7-eb874f124845 admin admin] PATH_INFO: `/v3/projects/4c31e6c9ef2c432386ba8e2a0a80b408/users/6070a5fec40348a589c73a7fc8791a52/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:03.891756 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f1a1550-0af6-47e9-a1e7-eb874f124845 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4c31e6c9ef2c432386ba8e2a0a80b408, user_id=6070a5fec40348a589c73a7fc8791a52, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:03.894212 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f1a1550-0af6-47e9-a1e7-eb874f124845 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:03.909561 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1059/2117] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:54:03 2022] PUT /identity/v3/projects/4c31e6c9ef2c432386ba8e2a0a80b408/users/6070a5fec40348a589c73a7fc8791a52/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:54:03.913598 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac7f6a7c-a4a0-433a-a985-e7df4a781691 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:03.934271 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac7f6a7c-a4a0-433a-a985-e7df4a781691 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:04.061780 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ebcd67-9806-4d76-885b-911306964cb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:04.062680 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ebcd67-9806-4d76-885b-911306964cb3 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:04.063051 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ebcd67-9806-4d76-885b-911306964cb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:04.063385 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ebcd67-9806-4d76-885b-911306964cb3 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:04.064101 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91ebcd67-9806-4d76-885b-911306964cb3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:04.071014 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91ebcd67-9806-4d76-885b-911306964cb3 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:04.079839 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1059/2118] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:54:03 2022] GET /identity/v3/roles => generated 1420 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 11 00:54:04.086699 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1a1550-0af6-47e9-a1e7-eb874f124845 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:04.101317 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1a1550-0af6-47e9-a1e7-eb874f124845 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:04.115049 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0478f0e-cf0c-4571-acbe-01f225ff2efa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:04.115841 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0478f0e-cf0c-4571-acbe-01f225ff2efa admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:04.116177 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0478f0e-cf0c-4571-acbe-01f225ff2efa admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:04.116501 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0478f0e-cf0c-4571-acbe-01f225ff2efa admin admin] PATH_INFO: `/v3/projects/4c31e6c9ef2c432386ba8e2a0a80b408/users/6070a5fec40348a589c73a7fc8791a52/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:04.118627 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0478f0e-cf0c-4571-acbe-01f225ff2efa admin admin] RBAC: Authorizing `identity:create_grant(project_id=4c31e6c9ef2c432386ba8e2a0a80b408, user_id=6070a5fec40348a589c73a7fc8791a52, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:04.122487 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0478f0e-cf0c-4571-acbe-01f225ff2efa admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:04.142620 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1060/2119] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:54:04 2022] PUT /identity/v3/projects/4c31e6c9ef2c432386ba8e2a0a80b408/users/6070a5fec40348a589c73a7fc8791a52/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:54:04.145403 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d2d689a-9720-49e4-af82-3f9232021b86 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:04.146230 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d2d689a-9720-49e4-af82-3f9232021b86 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:04.146654 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d2d689a-9720-49e4-af82-3f9232021b86 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:04.199371 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-6d2d689a-9720-49e4-af82-3f9232021b86 None None] MFA Rules not processed for user `6070a5fec40348a589c73a7fc8791a52`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:04.201071 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-6d2d689a-9720-49e4-af82-3f9232021b86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:04.332391 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1060/2120] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:04 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:54:04.344733 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0478f0e-cf0c-4571-acbe-01f225ff2efa admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:04.350693 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-e0478f0e-cf0c-4571-acbe-01f225ff2efa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:04.417996 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0478f0e-cf0c-4571-acbe-01f225ff2efa admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:04.421755 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772b9c7f-2bb9-44b2-bd7d-6de7704f1d38 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:04.422366 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772b9c7f-2bb9-44b2-bd7d-6de7704f1d38 service cinder] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:04.422643 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772b9c7f-2bb9-44b2-bd7d-6de7704f1d38 service cinder] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:04.422950 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772b9c7f-2bb9-44b2-bd7d-6de7704f1d38 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:04.435730 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-772b9c7f-2bb9-44b2-bd7d-6de7704f1d38 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:04.437691 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-772b9c7f-2bb9-44b2-bd7d-6de7704f1d38 service cinder] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:04.448582 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1061/2121] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:54:04 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:54:05.033750 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d2d689a-9720-49e4-af82-3f9232021b86 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:05.035980 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-6d2d689a-9720-49e4-af82-3f9232021b86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:05.049383 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d2d689a-9720-49e4-af82-3f9232021b86 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:05.054482 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961ce616-cd40-4bbc-a539-125fc55693a4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:05.055277 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961ce616-cd40-4bbc-a539-125fc55693a4 service cinder] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:05.055765 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961ce616-cd40-4bbc-a539-125fc55693a4 service cinder] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:05.056211 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961ce616-cd40-4bbc-a539-125fc55693a4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:05.081118 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:05.164094 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-961ce616-cd40-4bbc-a539-125fc55693a4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:05.166153 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-961ce616-cd40-4bbc-a539-125fc55693a4 service cinder] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:05.178806 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1061/2122] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:54:05 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:54:05.875458 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772b9c7f-2bb9-44b2-bd7d-6de7704f1d38 service cinder] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:05.876963 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-772b9c7f-2bb9-44b2-bd7d-6de7704f1d38 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:05.929902 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:05.944657 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772b9c7f-2bb9-44b2-bd7d-6de7704f1d38 service cinder] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:05.949357 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c68936ed-b60e-4e92-b48c-85d5c17042f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:05.949982 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c68936ed-b60e-4e92-b48c-85d5c17042f6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:05.950256 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c68936ed-b60e-4e92-b48c-85d5c17042f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:05.950544 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c68936ed-b60e-4e92-b48c-85d5c17042f6 admin admin] PATH_INFO: `/v3/users/8b1fc2c958f045538fb15d7d1d4a408a` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:05.951711 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c68936ed-b60e-4e92-b48c-85d5c17042f6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8b1fc2c958f045538fb15d7d1d4a408a)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:05.953667 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c68936ed-b60e-4e92-b48c-85d5c17042f6 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:06.003211 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-c68936ed-b60e-4e92-b48c-85d5c17042f6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8b1fc2c958f045538fb15d7d1d4a408a'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:06.003848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-c68936ed-b60e-4e92-b48c-85d5c17042f6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8b1fc2c958f045538fb15d7d1d4a408a'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:06.005113 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-c68936ed-b60e-4e92-b48c-85d5c17042f6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8b1fc2c958f045538fb15d7d1d4a408a'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:06.013438 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-c68936ed-b60e-4e92-b48c-85d5c17042f6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8b1fc2c958f045538fb15d7d1d4a408a'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:06.030089 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-c68936ed-b60e-4e92-b48c-85d5c17042f6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8b1fc2c958f045538fb15d7d1d4a408a'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:06.044757 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1062/2123] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:54:05 2022] DELETE /identity/v3/users/8b1fc2c958f045538fb15d7d1d4a408a => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:54:06.225828 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961ce616-cd40-4bbc-a539-125fc55693a4 service cinder] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:06.238805 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961ce616-cd40-4bbc-a539-125fc55693a4 service cinder] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:06.299006 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7e1965-9387-4561-867c-82fb9a871345 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:06.299637 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7e1965-9387-4561-867c-82fb9a871345 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:06.299918 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7e1965-9387-4561-867c-82fb9a871345 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:06.300190 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7e1965-9387-4561-867c-82fb9a871345 admin admin] PATH_INFO: `/v3/projects/832fdbe87ca847f094c6cc8a1fc43a59` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:06.301314 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a7e1965-9387-4561-867c-82fb9a871345 admin admin] RBAC: Authorizing `identity:delete_project(project_id=832fdbe87ca847f094c6cc8a1fc43a59)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:06.303274 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a7e1965-9387-4561-867c-82fb9a871345 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:06.353467 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-9a7e1965-9387-4561-867c-82fb9a871345 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '832fdbe87ca847f094c6cc8a1fc43a59', 'request_id': 'req-9a7e1965-9387-4561-867c-82fb9a871345'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:06.360248 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-9a7e1965-9387-4561-867c-82fb9a871345 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '832fdbe87ca847f094c6cc8a1fc43a59', 'request_id': 'req-9a7e1965-9387-4561-867c-82fb9a871345'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:06.367951 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-9a7e1965-9387-4561-867c-82fb9a871345 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '832fdbe87ca847f094c6cc8a1fc43a59', 'request_id': 'req-9a7e1965-9387-4561-867c-82fb9a871345'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:06.378881 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-9a7e1965-9387-4561-867c-82fb9a871345 admin admin] The token cache is being invalidate because project 832fdbe87ca847f094c6cc8a1fc43a59 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87924) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:54:06.379102 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-9a7e1965-9387-4561-867c-82fb9a871345 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:06.379746 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-9a7e1965-9387-4561-867c-82fb9a871345 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:06.381253 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1062/2124] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:54:06 2022] DELETE /identity/v3/projects/832fdbe87ca847f094c6cc8a1fc43a59 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:54:06.392818 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7714f89b-0994-4524-ac16-b9458eb323a4 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:06.393331 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7714f89b-0994-4524-ac16-b9458eb323a4 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:06.393757 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7714f89b-0994-4524-ac16-b9458eb323a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:06.416338 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-7714f89b-0994-4524-ac16-b9458eb323a4 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:06.417974 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-7714f89b-0994-4524-ac16-b9458eb323a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:06.479697 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1063/2125] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:54:06.487325 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d089279-d2c3-4eb0-8325-3fee8fa17ffa None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:06.487611 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d089279-d2c3-4eb0-8325-3fee8fa17ffa None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:06.487884 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d089279-d2c3-4eb0-8325-3fee8fa17ffa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:06.511525 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-0d089279-d2c3-4eb0-8325-3fee8fa17ffa None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:06.512763 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-0d089279-d2c3-4eb0-8325-3fee8fa17ffa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:06.520920 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1063/2126] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:54:06.525123 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7714f89b-0994-4524-ac16-b9458eb323a4 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:06.535927 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7714f89b-0994-4524-ac16-b9458eb323a4 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:06.540720 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8202ebe5-231d-462c-9647-2a9a38915e4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:06.541287 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8202ebe5-231d-462c-9647-2a9a38915e4c admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:06.541573 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8202ebe5-231d-462c-9647-2a9a38915e4c admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:06.541844 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8202ebe5-231d-462c-9647-2a9a38915e4c admin admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:06.542734 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8202ebe5-231d-462c-9647-2a9a38915e4c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:54:06.543121 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8202ebe5-231d-462c-9647-2a9a38915e4c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:06.544803 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8202ebe5-231d-462c-9647-2a9a38915e4c admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:06.556613 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1064/2127] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 00:54:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:54:06.561072 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d089279-d2c3-4eb0-8325-3fee8fa17ffa None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:06.570343 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d089279-d2c3-4eb0-8325-3fee8fa17ffa None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:06.575635 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea5547a8-917c-48ce-9dfa-10a2288b6056 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:06.576226 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5547a8-917c-48ce-9dfa-10a2288b6056 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:06.576503 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5547a8-917c-48ce-9dfa-10a2288b6056 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:06.576773 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5547a8-917c-48ce-9dfa-10a2288b6056 admin admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:06.577409 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea5547a8-917c-48ce-9dfa-10a2288b6056 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:06.579355 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea5547a8-917c-48ce-9dfa-10a2288b6056 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:06.606011 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1064/2128] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:54:06 2022] POST /identity/v3/projects => generated 375 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:54:06.610102 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8202ebe5-231d-462c-9647-2a9a38915e4c admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:06.620063 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8202ebe5-231d-462c-9647-2a9a38915e4c admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:06.691193 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f054b8-68de-4539-8712-3f2b76f305f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:06.691802 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f054b8-68de-4539-8712-3f2b76f305f5 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:06.692081 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f054b8-68de-4539-8712-3f2b76f305f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:06.692343 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f054b8-68de-4539-8712-3f2b76f305f5 admin admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:06.692964 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00f054b8-68de-4539-8712-3f2b76f305f5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:06.694889 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00f054b8-68de-4539-8712-3f2b76f305f5 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:06.703704 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:06.725913 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1065/2129] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:54:06 2022] POST /identity/v3/users => generated 288 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:54:06.727226 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea5547a8-917c-48ce-9dfa-10a2288b6056 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:06.737960 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea5547a8-917c-48ce-9dfa-10a2288b6056 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:06.742651 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c98df6a-d6c6-4509-8608-71ab4ed7288b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:06.743294 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c98df6a-d6c6-4509-8608-71ab4ed7288b admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:06.743570 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c98df6a-d6c6-4509-8608-71ab4ed7288b admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:06.743841 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c98df6a-d6c6-4509-8608-71ab4ed7288b admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:06.744514 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c98df6a-d6c6-4509-8608-71ab4ed7288b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:06.746823 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c98df6a-d6c6-4509-8608-71ab4ed7288b admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:06.757120 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1065/2130] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:54:06 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:54:06.761932 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f054b8-68de-4539-8712-3f2b76f305f5 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:06.773179 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f054b8-68de-4539-8712-3f2b76f305f5 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:06.778296 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e6aa85d-834e-4d39-ac82-d1ffd4e0ca3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:06.778933 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6aa85d-834e-4d39-ac82-d1ffd4e0ca3b admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:06.779211 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6aa85d-834e-4d39-ac82-d1ffd4e0ca3b admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:06.779483 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6aa85d-834e-4d39-ac82-d1ffd4e0ca3b admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:06.780131 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e6aa85d-834e-4d39-ac82-d1ffd4e0ca3b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:06.781737 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e6aa85d-834e-4d39-ac82-d1ffd4e0ca3b admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:06.793122 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1066/2131] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:54:06 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:54:06.797357 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c98df6a-d6c6-4509-8608-71ab4ed7288b admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:06.809878 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c98df6a-d6c6-4509-8608-71ab4ed7288b admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:06.815348 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664c9362-6076-49a2-9a3b-041f97884e6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:06.816015 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664c9362-6076-49a2-9a3b-041f97884e6f admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:06.816296 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664c9362-6076-49a2-9a3b-041f97884e6f admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:06.816569 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664c9362-6076-49a2-9a3b-041f97884e6f admin admin] PATH_INFO: `/v3/projects/31655ad33439478a9a9bf0582cc908a1/users/f5c7de0bf5b4441cb887eb8822a1bb71/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:06.820223 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:06.834232 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-664c9362-6076-49a2-9a3b-041f97884e6f admin admin] RBAC: Authorizing `identity:create_grant(project_id=31655ad33439478a9a9bf0582cc908a1, user_id=f5c7de0bf5b4441cb887eb8822a1bb71, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:06.837075 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-664c9362-6076-49a2-9a3b-041f97884e6f admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:06.849790 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1066/2132] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:54:06 2022] PUT /identity/v3/projects/31655ad33439478a9a9bf0582cc908a1/users/f5c7de0bf5b4441cb887eb8822a1bb71/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:54:06.860195 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57f3978-89ee-4252-9ef2-18b3c260bf4e None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:06.860478 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57f3978-89ee-4252-9ef2-18b3c260bf4e None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:06.860752 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57f3978-89ee-4252-9ef2-18b3c260bf4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:06.889702 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-b57f3978-89ee-4252-9ef2-18b3c260bf4e None None] MFA Rules not processed for user `f5c7de0bf5b4441cb887eb8822a1bb71`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:06.890982 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-b57f3978-89ee-4252-9ef2-18b3c260bf4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:06.959246 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1067/2133] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:06 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:54:06.972435 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664c9362-6076-49a2-9a3b-041f97884e6f admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:06.974417 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-664c9362-6076-49a2-9a3b-041f97884e6f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:07.052955 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664c9362-6076-49a2-9a3b-041f97884e6f admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:07.059528 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81dc7402-3db5-4f1e-aa3c-2772a5458e16 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '835def0e6c6d4365a2767051509eb0d6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'ResellerAdmin', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:07.060122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81dc7402-3db5-4f1e-aa3c-2772a5458e16 service nova] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:07.060396 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81dc7402-3db5-4f1e-aa3c-2772a5458e16 service nova] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:07.060664 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81dc7402-3db5-4f1e-aa3c-2772a5458e16 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:07.068589 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81dc7402-3db5-4f1e-aa3c-2772a5458e16 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:07.070162 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81dc7402-3db5-4f1e-aa3c-2772a5458e16 service nova] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:07.083061 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1067/2134] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:54:06 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:54:07.310158 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b57f3978-89ee-4252-9ef2-18b3c260bf4e None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:07.387134 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b57f3978-89ee-4252-9ef2-18b3c260bf4e None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:07.391861 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc3c4d35-25d3-4a35-9ec1-5b00649c71c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:07.392446 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3c4d35-25d3-4a35-9ec1-5b00649c71c3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:07.392709 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3c4d35-25d3-4a35-9ec1-5b00649c71c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:07.392983 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3c4d35-25d3-4a35-9ec1-5b00649c71c3 admin admin] PATH_INFO: `/v3/users/f5c7de0bf5b4441cb887eb8822a1bb71` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:07.394110 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc3c4d35-25d3-4a35-9ec1-5b00649c71c3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f5c7de0bf5b4441cb887eb8822a1bb71)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:07.396228 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc3c4d35-25d3-4a35-9ec1-5b00649c71c3 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:07.444836 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-fc3c4d35-25d3-4a35-9ec1-5b00649c71c3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f5c7de0bf5b4441cb887eb8822a1bb71'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:07.445494 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-fc3c4d35-25d3-4a35-9ec1-5b00649c71c3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f5c7de0bf5b4441cb887eb8822a1bb71'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:07.446091 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-fc3c4d35-25d3-4a35-9ec1-5b00649c71c3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f5c7de0bf5b4441cb887eb8822a1bb71'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:07.454236 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-fc3c4d35-25d3-4a35-9ec1-5b00649c71c3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f5c7de0bf5b4441cb887eb8822a1bb71'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:07.469637 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:INSERT=3,keystone:SELECT=89,keystone:DELETE=8 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:07.472358 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-fc3c4d35-25d3-4a35-9ec1-5b00649c71c3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f5c7de0bf5b4441cb887eb8822a1bb71'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:07.488167 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1068/2135] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:54:07 2022] DELETE /identity/v3/users/f5c7de0bf5b4441cb887eb8822a1bb71 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:54:07.497655 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a01b90f-efa2-4c0e-8372-267c5081ee3c None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:07.498132 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a01b90f-efa2-4c0e-8372-267c5081ee3c None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:07.498253 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a01b90f-efa2-4c0e-8372-267c5081ee3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:07.517591 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-7a01b90f-efa2-4c0e-8372-267c5081ee3c None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:07.518809 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-7a01b90f-efa2-4c0e-8372-267c5081ee3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:07.528581 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc3c4d35-25d3-4a35-9ec1-5b00649c71c3 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:07.530050 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-fc3c4d35-25d3-4a35-9ec1-5b00649c71c3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:07.585418 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1068/2136] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:54:07.595456 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a01b90f-efa2-4c0e-8372-267c5081ee3c None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:07.596792 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-7a01b90f-efa2-4c0e-8372-267c5081ee3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:07.601048 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc3c4d35-25d3-4a35-9ec1-5b00649c71c3 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:07.615504 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-170b2b08-dd41-484e-9f8a-5c522e55315d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:07.616169 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-170b2b08-dd41-484e-9f8a-5c522e55315d admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:07.616450 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-170b2b08-dd41-484e-9f8a-5c522e55315d admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:07.616725 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-170b2b08-dd41-484e-9f8a-5c522e55315d admin admin] PATH_INFO: `/v3/users/589b5f57ac6c4838ae117daf3aa2423d` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:07.618025 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-170b2b08-dd41-484e-9f8a-5c522e55315d admin admin] RBAC: Authorizing `identity:delete_user(user_id=589b5f57ac6c4838ae117daf3aa2423d)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:07.619998 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-170b2b08-dd41-484e-9f8a-5c522e55315d admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:07.657600 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a01b90f-efa2-4c0e-8372-267c5081ee3c None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:07.661414 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ed8647-d98e-4296-92d8-faff92df2da1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:07.662015 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ed8647-d98e-4296-92d8-faff92df2da1 service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:07.662290 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ed8647-d98e-4296-92d8-faff92df2da1 service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:07.662559 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ed8647-d98e-4296-92d8-faff92df2da1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:07.669100 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-170b2b08-dd41-484e-9f8a-5c522e55315d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '589b5f57ac6c4838ae117daf3aa2423d'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:07.670126 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-170b2b08-dd41-484e-9f8a-5c522e55315d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '589b5f57ac6c4838ae117daf3aa2423d'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:07.671341 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-170b2b08-dd41-484e-9f8a-5c522e55315d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '589b5f57ac6c4838ae117daf3aa2423d'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:07.672659 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ed8647-d98e-4296-92d8-faff92df2da1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:07.674072 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ed8647-d98e-4296-92d8-faff92df2da1 service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:07.678420 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-170b2b08-dd41-484e-9f8a-5c522e55315d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '589b5f57ac6c4838ae117daf3aa2423d'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:07.683727 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1069/2137] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:54:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 11 00:54:07.692249 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-170b2b08-dd41-484e-9f8a-5c522e55315d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '589b5f57ac6c4838ae117daf3aa2423d'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:07.708745 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1069/2138] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:54:07 2022] DELETE /identity/v3/users/589b5f57ac6c4838ae117daf3aa2423d => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:54:07.713211 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d72c51-6fed-486f-9cc9-caac8a99c9a9 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:07.713559 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d72c51-6fed-486f-9cc9-caac8a99c9a9 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:07.714524 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d72c51-6fed-486f-9cc9-caac8a99c9a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:07.738630 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-74d72c51-6fed-486f-9cc9-caac8a99c9a9 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:07.739919 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-74d72c51-6fed-486f-9cc9-caac8a99c9a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:07.751289 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:07.824232 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1070/2139] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:54:07.836267 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-170b2b08-dd41-484e-9f8a-5c522e55315d admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:07.838237 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-170b2b08-dd41-484e-9f8a-5c522e55315d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:07.935650 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-170b2b08-dd41-484e-9f8a-5c522e55315d admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:07.942326 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21dfe53f-31ed-4b74-87ab-16177634669c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:07.943118 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21dfe53f-31ed-4b74-87ab-16177634669c service neutron] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:07.943473 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21dfe53f-31ed-4b74-87ab-16177634669c service neutron] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:07.943854 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21dfe53f-31ed-4b74-87ab-16177634669c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:07.955615 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21dfe53f-31ed-4b74-87ab-16177634669c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:07.957474 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21dfe53f-31ed-4b74-87ab-16177634669c service neutron] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:07.969840 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1070/2140] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:54:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:54:08.142655 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d72c51-6fed-486f-9cc9-caac8a99c9a9 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:08.144869 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-74d72c51-6fed-486f-9cc9-caac8a99c9a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:08.157476 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d72c51-6fed-486f-9cc9-caac8a99c9a9 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:08.165094 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a7bc304-0ef9-4fe1-b8b9-e24b7362886b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:08.165738 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7bc304-0ef9-4fe1-b8b9-e24b7362886b admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:08.166005 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7bc304-0ef9-4fe1-b8b9-e24b7362886b admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:08.166276 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7bc304-0ef9-4fe1-b8b9-e24b7362886b admin admin] PATH_INFO: `/v3/projects/31655ad33439478a9a9bf0582cc908a1` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:08.168188 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a7bc304-0ef9-4fe1-b8b9-e24b7362886b admin admin] RBAC: Authorizing `identity:delete_project(project_id=31655ad33439478a9a9bf0582cc908a1)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:08.170345 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a7bc304-0ef9-4fe1-b8b9-e24b7362886b admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:08.221572 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-2a7bc304-0ef9-4fe1-b8b9-e24b7362886b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '31655ad33439478a9a9bf0582cc908a1', 'request_id': 'req-2a7bc304-0ef9-4fe1-b8b9-e24b7362886b'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:08.228248 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-2a7bc304-0ef9-4fe1-b8b9-e24b7362886b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '31655ad33439478a9a9bf0582cc908a1', 'request_id': 'req-2a7bc304-0ef9-4fe1-b8b9-e24b7362886b'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:08.239097 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-2a7bc304-0ef9-4fe1-b8b9-e24b7362886b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '31655ad33439478a9a9bf0582cc908a1', 'request_id': 'req-2a7bc304-0ef9-4fe1-b8b9-e24b7362886b'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:08.252643 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-2a7bc304-0ef9-4fe1-b8b9-e24b7362886b admin admin] The token cache is being invalidate because project 31655ad33439478a9a9bf0582cc908a1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87924) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:54:08.252926 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-2a7bc304-0ef9-4fe1-b8b9-e24b7362886b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:08.253564 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-2a7bc304-0ef9-4fe1-b8b9-e24b7362886b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:08.255123 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1071/2141] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:54:08 2022] DELETE /identity/v3/projects/31655ad33439478a9a9bf0582cc908a1 => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:54:08.260894 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28093da7-d9f8-4fc5-a3a8-f325548a6860 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:08.261185 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28093da7-d9f8-4fc5-a3a8-f325548a6860 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:08.261504 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28093da7-d9f8-4fc5-a3a8-f325548a6860 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:08.281323 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-28093da7-d9f8-4fc5-a3a8-f325548a6860 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:08.282504 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-28093da7-d9f8-4fc5-a3a8-f325548a6860 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:08.287004 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:DELETE=12,keystone:SELECT=86,keystone:INSERT=2 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:08.343445 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1071/2142] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:54:08.352444 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334e5d71-d4c1-413c-a214-0628e7dca80d None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:08.352727 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334e5d71-d4c1-413c-a214-0628e7dca80d None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:08.353217 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334e5d71-d4c1-413c-a214-0628e7dca80d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:08.365755 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28093da7-d9f8-4fc5-a3a8-f325548a6860 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:08.368057 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-28093da7-d9f8-4fc5-a3a8-f325548a6860 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:08.377145 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-334e5d71-d4c1-413c-a214-0628e7dca80d None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:08.378341 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-334e5d71-d4c1-413c-a214-0628e7dca80d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:08.381250 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28093da7-d9f8-4fc5-a3a8-f325548a6860 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:08.391190 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8057f25c-7b6a-4875-8edf-bc5c9f9b3e49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:08.391834 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8057f25c-7b6a-4875-8edf-bc5c9f9b3e49 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:08.392518 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8057f25c-7b6a-4875-8edf-bc5c9f9b3e49 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:08.392518 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8057f25c-7b6a-4875-8edf-bc5c9f9b3e49 admin admin] PATH_INFO: `/v3/projects/7beb504f4bc7405589196a3a532f38c8` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:08.393591 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1072/2143] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:54:08.393853 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8057f25c-7b6a-4875-8edf-bc5c9f9b3e49 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7beb504f4bc7405589196a3a532f38c8)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:08.395694 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8057f25c-7b6a-4875-8edf-bc5c9f9b3e49 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:08.398334 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-334e5d71-d4c1-413c-a214-0628e7dca80d None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:08.408253 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-334e5d71-d4c1-413c-a214-0628e7dca80d None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:08.413118 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d415226f-57c2-4b28-8b23-9709fffccd2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:08.413714 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d415226f-57c2-4b28-8b23-9709fffccd2d admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:08.413990 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d415226f-57c2-4b28-8b23-9709fffccd2d admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:08.414255 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d415226f-57c2-4b28-8b23-9709fffccd2d admin admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:08.414950 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d415226f-57c2-4b28-8b23-9709fffccd2d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:54:08.415249 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d415226f-57c2-4b28-8b23-9709fffccd2d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:08.416827 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d415226f-57c2-4b28-8b23-9709fffccd2d admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:08.431989 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1073/2144] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 00:54:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:54:08.433984 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d415226f-57c2-4b28-8b23-9709fffccd2d admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:08.444596 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d415226f-57c2-4b28-8b23-9709fffccd2d admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:08.455378 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-8057f25c-7b6a-4875-8edf-bc5c9f9b3e49 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7beb504f4bc7405589196a3a532f38c8', 'request_id': 'req-8057f25c-7b6a-4875-8edf-bc5c9f9b3e49'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:08.462102 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-8057f25c-7b6a-4875-8edf-bc5c9f9b3e49 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7beb504f4bc7405589196a3a532f38c8', 'request_id': 'req-8057f25c-7b6a-4875-8edf-bc5c9f9b3e49'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:08.470967 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-8057f25c-7b6a-4875-8edf-bc5c9f9b3e49 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7beb504f4bc7405589196a3a532f38c8', 'request_id': 'req-8057f25c-7b6a-4875-8edf-bc5c9f9b3e49'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:08.485134 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-8057f25c-7b6a-4875-8edf-bc5c9f9b3e49 admin admin] The token cache is being invalidate because project 7beb504f4bc7405589196a3a532f38c8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87923) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:54:08.485425 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-8057f25c-7b6a-4875-8edf-bc5c9f9b3e49 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:08.486077 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-8057f25c-7b6a-4875-8edf-bc5c9f9b3e49 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:08.488261 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1072/2145] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:54:08 2022] DELETE /identity/v3/projects/7beb504f4bc7405589196a3a532f38c8 => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:54:08.493012 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895d9008-d92c-46a5-8170-cf493dbd744a None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:08.493647 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895d9008-d92c-46a5-8170-cf493dbd744a None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:08.493950 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895d9008-d92c-46a5-8170-cf493dbd744a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:08.505449 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da83d7bd-92f6-44e9-a82d-1679dc99a32a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:08.506231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da83d7bd-92f6-44e9-a82d-1679dc99a32a admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:08.506512 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da83d7bd-92f6-44e9-a82d-1679dc99a32a admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:08.506983 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da83d7bd-92f6-44e9-a82d-1679dc99a32a admin admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:08.507616 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da83d7bd-92f6-44e9-a82d-1679dc99a32a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:08.509523 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da83d7bd-92f6-44e9-a82d-1679dc99a32a admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:08.515269 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-895d9008-d92c-46a5-8170-cf493dbd744a None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:08.516524 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-895d9008-d92c-46a5-8170-cf493dbd744a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:08.525299 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1073/2146] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:54:08.532967 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b50796-9191-4791-b67e-a0a0116699f7 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:08.533424 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b50796-9191-4791-b67e-a0a0116699f7 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:08.533699 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b50796-9191-4791-b67e-a0a0116699f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:08.542009 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1074/2147] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:54:08 2022] POST /identity/v3/projects => generated 383 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:54:08.547241 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da83d7bd-92f6-44e9-a82d-1679dc99a32a admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:08.549289 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-da83d7bd-92f6-44e9-a82d-1679dc99a32a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:08.557083 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-a7b50796-9191-4791-b67e-a0a0116699f7 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:08.558170 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:08.559248 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-a7b50796-9191-4791-b67e-a0a0116699f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:08.619144 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1074/2148] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:54:08.623834 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b50796-9191-4791-b67e-a0a0116699f7 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:08.624300 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da83d7bd-92f6-44e9-a82d-1679dc99a32a admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:08.628960 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51907fd1-3016-4425-ad77-342fd420888a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:08.629564 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51907fd1-3016-4425-ad77-342fd420888a admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:08.629841 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51907fd1-3016-4425-ad77-342fd420888a admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:08.630113 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51907fd1-3016-4425-ad77-342fd420888a admin admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:08.630736 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51907fd1-3016-4425-ad77-342fd420888a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:08.632421 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b50796-9191-4791-b67e-a0a0116699f7 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:08.632664 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51907fd1-3016-4425-ad77-342fd420888a admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:08.636992 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92643ab-866b-48d2-8a1e-76e33e4c9453 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:08.637627 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92643ab-866b-48d2-8a1e-76e33e4c9453 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:08.637903 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92643ab-866b-48d2-8a1e-76e33e4c9453 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:08.638169 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92643ab-866b-48d2-8a1e-76e33e4c9453 admin admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:08.638823 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c92643ab-866b-48d2-8a1e-76e33e4c9453 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:54:08.639125 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c92643ab-866b-48d2-8a1e-76e33e4c9453 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:08.640702 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c92643ab-866b-48d2-8a1e-76e33e4c9453 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:08.653218 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1075/2149] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 00:54:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:54:08.656625 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1075/2150] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:54:08 2022] POST /identity/v3/users => generated 292 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:54:08.658438 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51907fd1-3016-4425-ad77-342fd420888a admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:08.658551 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92643ab-866b-48d2-8a1e-76e33e4c9453 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:08.669313 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51907fd1-3016-4425-ad77-342fd420888a admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:08.670155 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92643ab-866b-48d2-8a1e-76e33e4c9453 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:08.673930 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d9283b-b674-4185-a7a2-30e69c6f1ad7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:08.674506 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d9283b-b674-4185-a7a2-30e69c6f1ad7 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:08.674779 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d9283b-b674-4185-a7a2-30e69c6f1ad7 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:08.674909 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-645e8b89-e178-44f0-94c6-1c8e5df0ea2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:08.675076 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d9283b-b674-4185-a7a2-30e69c6f1ad7 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:08.675538 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-645e8b89-e178-44f0-94c6-1c8e5df0ea2d admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:08.675711 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61d9283b-b674-4185-a7a2-30e69c6f1ad7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:08.675811 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-645e8b89-e178-44f0-94c6-1c8e5df0ea2d admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:08.676074 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-645e8b89-e178-44f0-94c6-1c8e5df0ea2d admin admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:08.676698 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-645e8b89-e178-44f0-94c6-1c8e5df0ea2d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:08.677275 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61d9283b-b674-4185-a7a2-30e69c6f1ad7 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:08.678568 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-645e8b89-e178-44f0-94c6-1c8e5df0ea2d admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:08.688083 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1076/2151] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:54:08 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:54:08.691862 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d9283b-b674-4185-a7a2-30e69c6f1ad7 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:08.693089 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:08.702260 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d9283b-b674-4185-a7a2-30e69c6f1ad7 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:08.708141 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18ac6ee4-7d54-4c60-87fb-9a52ebad2361 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:08.708746 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ac6ee4-7d54-4c60-87fb-9a52ebad2361 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:08.709027 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ac6ee4-7d54-4c60-87fb-9a52ebad2361 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:08.709290 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ac6ee4-7d54-4c60-87fb-9a52ebad2361 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:08.709908 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18ac6ee4-7d54-4c60-87fb-9a52ebad2361 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:08.711509 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18ac6ee4-7d54-4c60-87fb-9a52ebad2361 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:08.714813 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1076/2152] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:54:08 2022] POST /identity/v3/projects => generated 377 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:54:08.717788 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-645e8b89-e178-44f0-94c6-1c8e5df0ea2d admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:08.728262 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1077/2153] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:54:08 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 11 00:54:08.729272 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18ac6ee4-7d54-4c60-87fb-9a52ebad2361 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:08.740027 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18ac6ee4-7d54-4c60-87fb-9a52ebad2361 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:08.781468 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-645e8b89-e178-44f0-94c6-1c8e5df0ea2d admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:08.786738 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86b840b5-6692-4d82-8738-db22d1278303 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:08.787365 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b840b5-6692-4d82-8738-db22d1278303 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:08.787644 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b840b5-6692-4d82-8738-db22d1278303 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:08.787932 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b840b5-6692-4d82-8738-db22d1278303 admin admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:08.788567 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86b840b5-6692-4d82-8738-db22d1278303 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:08.790517 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86b840b5-6692-4d82-8738-db22d1278303 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:08.797892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6357465b-9409-4947-9daf-ae81f4ed99e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:08.798557 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6357465b-9409-4947-9daf-ae81f4ed99e0 admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:08.798860 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6357465b-9409-4947-9daf-ae81f4ed99e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:08.799139 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6357465b-9409-4947-9daf-ae81f4ed99e0 admin admin] PATH_INFO: `/v3/projects/a558aced0de64bae905abf66d5fa8736/users/c86a75a8d8734d08901b026e1ffa0caa/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:08.812762 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1077/2154] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:54:08 2022] POST /identity/v3/users => generated 289 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:54:08.815071 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6357465b-9409-4947-9daf-ae81f4ed99e0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a558aced0de64bae905abf66d5fa8736, user_id=c86a75a8d8734d08901b026e1ffa0caa, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:08.816696 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86b840b5-6692-4d82-8738-db22d1278303 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:08.817727 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6357465b-9409-4947-9daf-ae81f4ed99e0 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:08.827682 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86b840b5-6692-4d82-8738-db22d1278303 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:08.831989 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1078/2155] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:54:08 2022] PUT /identity/v3/projects/a558aced0de64bae905abf66d5fa8736/users/c86a75a8d8734d08901b026e1ffa0caa/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:54:08.842845 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06dc1c05-00b4-4155-8977-9bad93df2e60 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:08.843132 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06dc1c05-00b4-4155-8977-9bad93df2e60 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:08.843405 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06dc1c05-00b4-4155-8977-9bad93df2e60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:08.872305 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-06dc1c05-00b4-4155-8977-9bad93df2e60 None None] MFA Rules not processed for user `c86a75a8d8734d08901b026e1ffa0caa`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:08.873526 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-06dc1c05-00b4-4155-8977-9bad93df2e60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:08.889146 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e35224fe-149d-47b5-9ecf-a8faa79776bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:08.889757 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35224fe-149d-47b5-9ecf-a8faa79776bb admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:08.890033 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35224fe-149d-47b5-9ecf-a8faa79776bb admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:08.890307 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35224fe-149d-47b5-9ecf-a8faa79776bb admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:08.890986 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e35224fe-149d-47b5-9ecf-a8faa79776bb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:08.892573 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e35224fe-149d-47b5-9ecf-a8faa79776bb admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:08.902383 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1078/2156] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:54:08 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:54:08.906202 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e35224fe-149d-47b5-9ecf-a8faa79776bb admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:08.916002 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e35224fe-149d-47b5-9ecf-a8faa79776bb admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:08.916148 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:08.922561 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d998eda4-df76-438c-87db-d5b784eeb684 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:08.923182 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d998eda4-df76-438c-87db-d5b784eeb684 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:08.923459 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d998eda4-df76-438c-87db-d5b784eeb684 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:08.923727 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d998eda4-df76-438c-87db-d5b784eeb684 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:08.924360 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d998eda4-df76-438c-87db-d5b784eeb684 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:08.925955 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d998eda4-df76-438c-87db-d5b784eeb684 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:08.940577 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1079/2157] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:54:08 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 11 00:54:08.941447 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1079/2158] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:08 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:54:08.943620 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d998eda4-df76-438c-87db-d5b784eeb684 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:08.945700 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06dc1c05-00b4-4155-8977-9bad93df2e60 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:08.954117 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d998eda4-df76-438c-87db-d5b784eeb684 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:08.958829 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06dc1c05-00b4-4155-8977-9bad93df2e60 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:08.961563 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f36ffb1c-a7f9-4a24-82e1-1f77a92b8c89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:08.961563 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36ffb1c-a7f9-4a24-82e1-1f77a92b8c89 admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:08.961563 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36ffb1c-a7f9-4a24-82e1-1f77a92b8c89 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:08.961563 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36ffb1c-a7f9-4a24-82e1-1f77a92b8c89 admin admin] PATH_INFO: `/v3/projects/7656f24070314d4480d607f85937a84d/users/f8aa86aad7d34cc2a707b2cdfe47e8a3/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:08.967351 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273b59cd-b49a-436e-8ef4-8aa1833090b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:08.967962 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273b59cd-b49a-436e-8ef4-8aa1833090b6 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:08.968239 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273b59cd-b49a-436e-8ef4-8aa1833090b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:08.968509 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273b59cd-b49a-436e-8ef4-8aa1833090b6 admin admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:08.969207 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-273b59cd-b49a-436e-8ef4-8aa1833090b6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:08.971227 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-273b59cd-b49a-436e-8ef4-8aa1833090b6 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:08.976323 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f36ffb1c-a7f9-4a24-82e1-1f77a92b8c89 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7656f24070314d4480d607f85937a84d, user_id=f8aa86aad7d34cc2a707b2cdfe47e8a3, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:08.979036 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f36ffb1c-a7f9-4a24-82e1-1f77a92b8c89 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:08.993130 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1080/2159] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:54:08 2022] PUT /identity/v3/projects/7656f24070314d4480d607f85937a84d/users/f8aa86aad7d34cc2a707b2cdfe47e8a3/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:54:09.004135 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1080/2160] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:54:08 2022] POST /identity/v3/projects => generated 381 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:54:09.004331 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9c967c-8c8e-47ad-99cd-2107022f8384 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:09.004590 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9c967c-8c8e-47ad-99cd-2107022f8384 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:09.004866 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9c967c-8c8e-47ad-99cd-2107022f8384 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:09.005047 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273b59cd-b49a-436e-8ef4-8aa1833090b6 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:09.046403 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-9a9c967c-8c8e-47ad-99cd-2107022f8384 None None] MFA Rules not processed for user `f8aa86aad7d34cc2a707b2cdfe47e8a3`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:09.047953 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-9a9c967c-8c8e-47ad-99cd-2107022f8384 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:09.072423 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273b59cd-b49a-436e-8ef4-8aa1833090b6 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:09.082560 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0206a40-8eb2-451d-ac81-462953a908a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:09.082735 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:09.082854 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0206a40-8eb2-451d-ac81-462953a908a4 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:09.082854 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0206a40-8eb2-451d-ac81-462953a908a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:09.082854 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0206a40-8eb2-451d-ac81-462953a908a4 admin admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:09.082854 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0206a40-8eb2-451d-ac81-462953a908a4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:09.082854 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0206a40-8eb2-451d-ac81-462953a908a4 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:09.104517 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1081/2161] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:54:09 2022] POST /identity/v3/users => generated 297 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:54:09.109476 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0206a40-8eb2-451d-ac81-462953a908a4 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:09.119130 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0206a40-8eb2-451d-ac81-462953a908a4 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:09.124752 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41aac8d0-af9f-4b7a-897e-e18b6e1f5e51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:09.124893 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1081/2162] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:09 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:54:09.127409 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41aac8d0-af9f-4b7a-897e-e18b6e1f5e51 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:09.127696 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41aac8d0-af9f-4b7a-897e-e18b6e1f5e51 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:09.128063 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41aac8d0-af9f-4b7a-897e-e18b6e1f5e51 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:09.128724 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41aac8d0-af9f-4b7a-897e-e18b6e1f5e51 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:09.129202 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a9c967c-8c8e-47ad-99cd-2107022f8384 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:09.130326 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41aac8d0-af9f-4b7a-897e-e18b6e1f5e51 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:09.139859 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a9c967c-8c8e-47ad-99cd-2107022f8384 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:09.142507 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1082/2163] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:54:09 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:54:09.144856 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f0d42d1-8cc7-449c-b0b6-da86569d3f90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:09.145220 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41aac8d0-af9f-4b7a-897e-e18b6e1f5e51 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:09.145454 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f0d42d1-8cc7-449c-b0b6-da86569d3f90 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:09.145837 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f0d42d1-8cc7-449c-b0b6-da86569d3f90 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:09.146089 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f0d42d1-8cc7-449c-b0b6-da86569d3f90 admin admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:09.146724 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f0d42d1-8cc7-449c-b0b6-da86569d3f90 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:09.148654 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f0d42d1-8cc7-449c-b0b6-da86569d3f90 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:09.156896 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41aac8d0-af9f-4b7a-897e-e18b6e1f5e51 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:09.163206 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8def110-2130-426b-a616-9578e229ad35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:09.163917 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8def110-2130-426b-a616-9578e229ad35 admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:09.164227 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8def110-2130-426b-a616-9578e229ad35 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:09.164501 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8def110-2130-426b-a616-9578e229ad35 admin admin] PATH_INFO: `/v3/projects/480066f74364450eb0e137b28c77e9f9/users/0d9f8b30237642d789f1e66d05478d3e/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:09.179962 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1082/2164] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:54:09 2022] POST /identity/v3/projects => generated 373 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:54:09.182064 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8def110-2130-426b-a616-9578e229ad35 admin admin] RBAC: Authorizing `identity:create_grant(project_id=480066f74364450eb0e137b28c77e9f9, user_id=0d9f8b30237642d789f1e66d05478d3e, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:09.183426 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f0d42d1-8cc7-449c-b0b6-da86569d3f90 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:09.184875 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8def110-2130-426b-a616-9578e229ad35 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:09.200587 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1083/2165] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:54:09 2022] PUT /identity/v3/projects/480066f74364450eb0e137b28c77e9f9/users/0d9f8b30237642d789f1e66d05478d3e/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:54:09.204809 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8def110-2130-426b-a616-9578e229ad35 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:09.254719 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f0d42d1-8cc7-449c-b0b6-da86569d3f90 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:09.274946 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8def110-2130-426b-a616-9578e229ad35 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:09.281999 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eb18052-ef3c-4ffc-a7ef-89a3a94dab93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:09.282817 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb18052-ef3c-4ffc-a7ef-89a3a94dab93 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:09.283062 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb18052-ef3c-4ffc-a7ef-89a3a94dab93 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:09.283429 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb18052-ef3c-4ffc-a7ef-89a3a94dab93 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:09.284527 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eb18052-ef3c-4ffc-a7ef-89a3a94dab93 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:09.286457 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eb18052-ef3c-4ffc-a7ef-89a3a94dab93 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:09.297594 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1084/2166] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:54:09 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:54:09.301505 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eb18052-ef3c-4ffc-a7ef-89a3a94dab93 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:09.312613 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eb18052-ef3c-4ffc-a7ef-89a3a94dab93 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:09.317914 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e923c9b2-085d-4c63-a04d-8ee0373f8f5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:09.318695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e923c9b2-085d-4c63-a04d-8ee0373f8f5f admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:09.319014 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e923c9b2-085d-4c63-a04d-8ee0373f8f5f admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:09.319297 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e923c9b2-085d-4c63-a04d-8ee0373f8f5f admin admin] PATH_INFO: `/v3/projects/480066f74364450eb0e137b28c77e9f9/users/0d9f8b30237642d789f1e66d05478d3e/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:09.321382 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e923c9b2-085d-4c63-a04d-8ee0373f8f5f admin admin] RBAC: Authorizing `identity:create_grant(project_id=480066f74364450eb0e137b28c77e9f9, user_id=0d9f8b30237642d789f1e66d05478d3e, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:09.324171 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e923c9b2-085d-4c63-a04d-8ee0373f8f5f admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:09.330043 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:09.330819 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62338aa8-61b3-40d0-b1fc-b0f10abd96f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:09.331496 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62338aa8-61b3-40d0-b1fc-b0f10abd96f9 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:09.331914 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62338aa8-61b3-40d0-b1fc-b0f10abd96f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:09.332311 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62338aa8-61b3-40d0-b1fc-b0f10abd96f9 admin admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:09.333190 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62338aa8-61b3-40d0-b1fc-b0f10abd96f9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:09.336197 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62338aa8-61b3-40d0-b1fc-b0f10abd96f9 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:09.344592 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1085/2167] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:54:09 2022] PUT /identity/v3/projects/480066f74364450eb0e137b28c77e9f9/users/0d9f8b30237642d789f1e66d05478d3e/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:54:09.348584 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4063965-9e9c-43b0-bc72-96b54d9e23cb None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:09.348897 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4063965-9e9c-43b0-bc72-96b54d9e23cb None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:09.349147 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4063965-9e9c-43b0-bc72-96b54d9e23cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:09.361503 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1083/2168] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:54:09 2022] POST /identity/v3/users => generated 287 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:54:09.362523 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62338aa8-61b3-40d0-b1fc-b0f10abd96f9 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:09.372271 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62338aa8-61b3-40d0-b1fc-b0f10abd96f9 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:09.379342 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-d4063965-9e9c-43b0-bc72-96b54d9e23cb None None] MFA Rules not processed for user `0d9f8b30237642d789f1e66d05478d3e`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:09.380561 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-d4063965-9e9c-43b0-bc72-96b54d9e23cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:09.421475 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:09.437254 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff2139f-4acc-491b-aa99-b9eac39113c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:09.437799 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff2139f-4acc-491b-aa99-b9eac39113c3 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:09.438075 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff2139f-4acc-491b-aa99-b9eac39113c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:09.438344 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff2139f-4acc-491b-aa99-b9eac39113c3 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:09.439022 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eff2139f-4acc-491b-aa99-b9eac39113c3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:09.440613 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eff2139f-4acc-491b-aa99-b9eac39113c3 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:09.451855 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1084/2169] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:54:09 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:54:09.459438 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1086/2170] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:09 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:54:09.459613 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff2139f-4acc-491b-aa99-b9eac39113c3 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:09.466306 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff2139f-4acc-491b-aa99-b9eac39113c3 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:09.467405 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4063965-9e9c-43b0-bc72-96b54d9e23cb None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:09.468791 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-d4063965-9e9c-43b0-bc72-96b54d9e23cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:09.471032 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dbde251-d4c1-4e97-a405-c34773b8850b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:09.471633 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbde251-d4c1-4e97-a405-c34773b8850b admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:09.471914 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbde251-d4c1-4e97-a405-c34773b8850b admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:09.472188 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbde251-d4c1-4e97-a405-c34773b8850b admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:09.472834 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dbde251-d4c1-4e97-a405-c34773b8850b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:09.474422 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dbde251-d4c1-4e97-a405-c34773b8850b admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:09.486895 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1085/2171] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:54:09 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 11 00:54:09.490953 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dbde251-d4c1-4e97-a405-c34773b8850b admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:09.500856 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dbde251-d4c1-4e97-a405-c34773b8850b admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:09.505453 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6633ab0-4dff-4e75-bb66-15ce7aba9829 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:09.506083 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6633ab0-4dff-4e75-bb66-15ce7aba9829 admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:09.506391 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6633ab0-4dff-4e75-bb66-15ce7aba9829 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:09.506666 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6633ab0-4dff-4e75-bb66-15ce7aba9829 admin admin] PATH_INFO: `/v3/projects/c2d6e7dabf6445539f5f783ec7991cee/users/a12f234cac6141dd99e62168c3368b3a/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:09.522197 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6633ab0-4dff-4e75-bb66-15ce7aba9829 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c2d6e7dabf6445539f5f783ec7991cee, user_id=a12f234cac6141dd99e62168c3368b3a, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:09.524859 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6633ab0-4dff-4e75-bb66-15ce7aba9829 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:09.531679 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4063965-9e9c-43b0-bc72-96b54d9e23cb None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:09.536058 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fef3299d-4f58-4595-83a7-1a6ee84474e3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:09.536663 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef3299d-4f58-4595-83a7-1a6ee84474e3 service cinder] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:09.536942 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef3299d-4f58-4595-83a7-1a6ee84474e3 service cinder] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:09.537211 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef3299d-4f58-4595-83a7-1a6ee84474e3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:09.540107 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1086/2172] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:54:09 2022] PUT /identity/v3/projects/c2d6e7dabf6445539f5f783ec7991cee/users/a12f234cac6141dd99e62168c3368b3a/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:54:09.550518 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fef3299d-4f58-4595-83a7-1a6ee84474e3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:09.552277 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-002d74b3-ad90-4fe4-a66f-fc0ae46ab677 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:09.552396 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fef3299d-4f58-4595-83a7-1a6ee84474e3 service cinder] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:09.552559 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-002d74b3-ad90-4fe4-a66f-fc0ae46ab677 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:09.552828 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-002d74b3-ad90-4fe4-a66f-fc0ae46ab677 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:09.562448 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1087/2173] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:54:09 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 11 00:54:09.582511 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-002d74b3-ad90-4fe4-a66f-fc0ae46ab677 None None] MFA Rules not processed for user `a12f234cac6141dd99e62168c3368b3a`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:09.583853 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-002d74b3-ad90-4fe4-a66f-fc0ae46ab677 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:09.669043 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1087/2174] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:09 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:54:09.673530 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fef3299d-4f58-4595-83a7-1a6ee84474e3 service cinder] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:09.685035 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fef3299d-4f58-4595-83a7-1a6ee84474e3 service cinder] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:09.688176 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-002d74b3-ad90-4fe4-a66f-fc0ae46ab677 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:09.689691 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-002d74b3-ad90-4fe4-a66f-fc0ae46ab677 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:09.742889 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd676749-e060-43c2-b0bb-30e4d9dfc8f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:09.743497 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd676749-e060-43c2-b0bb-30e4d9dfc8f0 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:09.743776 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd676749-e060-43c2-b0bb-30e4d9dfc8f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:09.744050 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd676749-e060-43c2-b0bb-30e4d9dfc8f0 admin admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:09.744677 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd676749-e060-43c2-b0bb-30e4d9dfc8f0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:09.746688 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd676749-e060-43c2-b0bb-30e4d9dfc8f0 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:09.749273 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-002d74b3-ad90-4fe4-a66f-fc0ae46ab677 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:09.752892 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d7bfb1-4e34-4d12-ab85-b09b42064571 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:09.753486 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d7bfb1-4e34-4d12-ab85-b09b42064571 service cinder] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:09.753756 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d7bfb1-4e34-4d12-ab85-b09b42064571 service cinder] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:09.754017 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d7bfb1-4e34-4d12-ab85-b09b42064571 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:09.775039 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1088/2175] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:54:09 2022] POST /identity/v3/projects => generated 375 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:54:09.779770 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd676749-e060-43c2-b0bb-30e4d9dfc8f0 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:09.792170 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:09.805000 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:09.816611 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76d7bfb1-4e34-4d12-ab85-b09b42064571 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:09.819388 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76d7bfb1-4e34-4d12-ab85-b09b42064571 service cinder] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:09.829309 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1088/2176] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:54:09 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:54:09.845441 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd676749-e060-43c2-b0bb-30e4d9dfc8f0 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:09.850084 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9476a98-44af-4404-91c1-2cf11e4df1da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:09.850685 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9476a98-44af-4404-91c1-2cf11e4df1da admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:09.851000 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9476a98-44af-4404-91c1-2cf11e4df1da admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:09.851273 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9476a98-44af-4404-91c1-2cf11e4df1da admin admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:09.851901 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9476a98-44af-4404-91c1-2cf11e4df1da admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:09.853864 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9476a98-44af-4404-91c1-2cf11e4df1da admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:09.877805 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1089/2177] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:54:09 2022] POST /identity/v3/users => generated 294 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:54:09.881835 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d7bfb1-4e34-4d12-ab85-b09b42064571 service cinder] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:09.893088 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d7bfb1-4e34-4d12-ab85-b09b42064571 service cinder] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:09.897736 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f60b7f-cfa6-4383-9123-508612d7b2ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:09.898327 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f60b7f-cfa6-4383-9123-508612d7b2ff admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:09.898600 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f60b7f-cfa6-4383-9123-508612d7b2ff admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:09.898895 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f60b7f-cfa6-4383-9123-508612d7b2ff admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:09.899530 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47f60b7f-cfa6-4383-9123-508612d7b2ff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:09.901130 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47f60b7f-cfa6-4383-9123-508612d7b2ff admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:09.912423 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1089/2178] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:54:09 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:54:09.916829 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9476a98-44af-4404-91c1-2cf11e4df1da admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:09.927144 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9476a98-44af-4404-91c1-2cf11e4df1da admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:09.931878 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a05f72-ecc3-48c4-a5e7-c92e968974e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:09.933034 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a05f72-ecc3-48c4-a5e7-c92e968974e0 admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:09.933378 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a05f72-ecc3-48c4-a5e7-c92e968974e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:09.933754 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a05f72-ecc3-48c4-a5e7-c92e968974e0 admin admin] PATH_INFO: `/v3/projects/7c878c184e224223b0a40568a615614a/users/25ef3d2ef08f4d85b60d923ec35ae092/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:09.950411 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11a05f72-ecc3-48c4-a5e7-c92e968974e0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7c878c184e224223b0a40568a615614a, user_id=25ef3d2ef08f4d85b60d923ec35ae092, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:09.953206 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11a05f72-ecc3-48c4-a5e7-c92e968974e0 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:09.966723 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1090/2179] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:54:09 2022] PUT /identity/v3/projects/7c878c184e224223b0a40568a615614a/users/25ef3d2ef08f4d85b60d923ec35ae092/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:54:09.970914 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f60b7f-cfa6-4383-9123-508612d7b2ff admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:09.982073 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f60b7f-cfa6-4383-9123-508612d7b2ff admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:10.043692 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc84f24d-2170-4f25-9c18-03d618ebcbcf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:10.044291 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc84f24d-2170-4f25-9c18-03d618ebcbcf admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:10.044572 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc84f24d-2170-4f25-9c18-03d618ebcbcf admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:10.044922 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc84f24d-2170-4f25-9c18-03d618ebcbcf admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:10.045637 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc84f24d-2170-4f25-9c18-03d618ebcbcf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:10.047421 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc84f24d-2170-4f25-9c18-03d618ebcbcf admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:10.062515 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1090/2180] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:54:09 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:54:10.063782 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a05f72-ecc3-48c4-a5e7-c92e968974e0 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:10.089605 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a05f72-ecc3-48c4-a5e7-c92e968974e0 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:10.095039 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eeda128-a380-4649-908d-3a7458e41232 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:10.095824 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eeda128-a380-4649-908d-3a7458e41232 admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:10.096198 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eeda128-a380-4649-908d-3a7458e41232 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:10.096572 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eeda128-a380-4649-908d-3a7458e41232 admin admin] PATH_INFO: `/v3/projects/7c878c184e224223b0a40568a615614a/users/25ef3d2ef08f4d85b60d923ec35ae092/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:10.098814 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eeda128-a380-4649-908d-3a7458e41232 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7c878c184e224223b0a40568a615614a, user_id=25ef3d2ef08f4d85b60d923ec35ae092, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:10.101849 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eeda128-a380-4649-908d-3a7458e41232 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:10.126836 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1091/2181] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:54:10 2022] PUT /identity/v3/projects/7c878c184e224223b0a40568a615614a/users/25ef3d2ef08f4d85b60d923ec35ae092/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:54:10.135060 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d9c02f-7395-4a11-b6cd-4bcbd6160acc None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:10.135348 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d9c02f-7395-4a11-b6cd-4bcbd6160acc None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:10.135623 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d9c02f-7395-4a11-b6cd-4bcbd6160acc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:10.165852 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-73d9c02f-7395-4a11-b6cd-4bcbd6160acc None None] MFA Rules not processed for user `25ef3d2ef08f4d85b60d923ec35ae092`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:10.167804 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-73d9c02f-7395-4a11-b6cd-4bcbd6160acc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:10.235101 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1091/2182] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:10 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:54:10.246838 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eeda128-a380-4649-908d-3a7458e41232 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:10.302026 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eeda128-a380-4649-908d-3a7458e41232 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:10.305696 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d41afe-00a8-4787-b54e-253e8e980391 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:10.306286 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d41afe-00a8-4787-b54e-253e8e980391 service cinder] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:10.306564 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d41afe-00a8-4787-b54e-253e8e980391 service cinder] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:10.306867 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d41afe-00a8-4787-b54e-253e8e980391 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:10.364453 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91d41afe-00a8-4787-b54e-253e8e980391 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:10.365939 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91d41afe-00a8-4787-b54e-253e8e980391 service cinder] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:10.375035 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1092/2183] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:54:10 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:54:11.925167 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d9c02f-7395-4a11-b6cd-4bcbd6160acc None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:11.935820 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d9c02f-7395-4a11-b6cd-4bcbd6160acc None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:11.939521 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f16675c-25a5-4cdd-b3c2-11d7e88e3dcc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:11.940110 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f16675c-25a5-4cdd-b3c2-11d7e88e3dcc service cinder] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:11.940387 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f16675c-25a5-4cdd-b3c2-11d7e88e3dcc service cinder] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:11.940661 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f16675c-25a5-4cdd-b3c2-11d7e88e3dcc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:12.000478 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f16675c-25a5-4cdd-b3c2-11d7e88e3dcc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:12.001915 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f16675c-25a5-4cdd-b3c2-11d7e88e3dcc service cinder] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:12.007773 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:12.013145 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1092/2184] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:54:11 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:54:12.878888 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d41afe-00a8-4787-b54e-253e8e980391 service cinder] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:12.888424 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d41afe-00a8-4787-b54e-253e8e980391 service cinder] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:12.892089 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3580a55-478a-4fc5-91a8-97295d94b59f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:12.892691 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3580a55-478a-4fc5-91a8-97295d94b59f service cinder] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:12.892972 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3580a55-478a-4fc5-91a8-97295d94b59f service cinder] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:12.893243 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3580a55-478a-4fc5-91a8-97295d94b59f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:12.902102 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3580a55-478a-4fc5-91a8-97295d94b59f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:12.903557 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3580a55-478a-4fc5-91a8-97295d94b59f service cinder] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:12.913160 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1093/2185] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:54:12 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:54:13.891963 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f16675c-25a5-4cdd-b3c2-11d7e88e3dcc service cinder] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:13.902272 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f16675c-25a5-4cdd-b3c2-11d7e88e3dcc service cinder] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:13.907661 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85682a54-537d-473e-bae0-53a786da279d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:13.908253 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85682a54-537d-473e-bae0-53a786da279d service cinder] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:13.908729 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85682a54-537d-473e-bae0-53a786da279d service cinder] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:13.908994 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85682a54-537d-473e-bae0-53a786da279d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:13.911748 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-85682a54-537d-473e-bae0-53a786da279d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:14.001738 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85682a54-537d-473e-bae0-53a786da279d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:14.003431 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85682a54-537d-473e-bae0-53a786da279d service cinder] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:14.017117 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1093/2186] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:54:13 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:54:14.900766 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3580a55-478a-4fc5-91a8-97295d94b59f service cinder] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:14.906243 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:14.987175 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3580a55-478a-4fc5-91a8-97295d94b59f service cinder] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:14.987175 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a8d423c-f2ad-4d0a-b0da-2e8f021f7db6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:14.987175 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8d423c-f2ad-4d0a-b0da-2e8f021f7db6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:14.987729 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8d423c-f2ad-4d0a-b0da-2e8f021f7db6 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:14.987829 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8d423c-f2ad-4d0a-b0da-2e8f021f7db6 admin admin] PATH_INFO: `/v3/users/c86a75a8d8734d08901b026e1ffa0caa` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:14.989001 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a8d423c-f2ad-4d0a-b0da-2e8f021f7db6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c86a75a8d8734d08901b026e1ffa0caa)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:14.991515 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a8d423c-f2ad-4d0a-b0da-2e8f021f7db6 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:15.045222 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-7a8d423c-f2ad-4d0a-b0da-2e8f021f7db6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c86a75a8d8734d08901b026e1ffa0caa'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:15.045909 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-7a8d423c-f2ad-4d0a-b0da-2e8f021f7db6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c86a75a8d8734d08901b026e1ffa0caa'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:15.046522 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-7a8d423c-f2ad-4d0a-b0da-2e8f021f7db6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c86a75a8d8734d08901b026e1ffa0caa'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:15.053842 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-7a8d423c-f2ad-4d0a-b0da-2e8f021f7db6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c86a75a8d8734d08901b026e1ffa0caa'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:15.066669 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-7a8d423c-f2ad-4d0a-b0da-2e8f021f7db6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c86a75a8d8734d08901b026e1ffa0caa'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:15.080609 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1094/2187] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:54:14 2022] DELETE /identity/v3/users/c86a75a8d8734d08901b026e1ffa0caa => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:54:15.086104 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9f95b6-ab91-4ce1-98ad-850426a2483e None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:15.086652 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9f95b6-ab91-4ce1-98ad-850426a2483e None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:15.086652 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9f95b6-ab91-4ce1-98ad-850426a2483e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:15.106054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-9a9f95b6-ab91-4ce1-98ad-850426a2483e None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:15.107281 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-9a9f95b6-ab91-4ce1-98ad-850426a2483e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:15.167047 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1094/2188] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:54:15.176862 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a8d423c-f2ad-4d0a-b0da-2e8f021f7db6 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:15.178255 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-7a8d423c-f2ad-4d0a-b0da-2e8f021f7db6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:15.238706 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a8d423c-f2ad-4d0a-b0da-2e8f021f7db6 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:15.242370 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5192deb6-f342-48b0-bbce-b8c3176faa5a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:15.242999 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5192deb6-f342-48b0-bbce-b8c3176faa5a service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:15.243275 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5192deb6-f342-48b0-bbce-b8c3176faa5a service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:15.243542 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5192deb6-f342-48b0-bbce-b8c3176faa5a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:15.252849 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5192deb6-f342-48b0-bbce-b8c3176faa5a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:15.254259 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5192deb6-f342-48b0-bbce-b8c3176faa5a service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:15.265415 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1095/2189] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:54:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:54:15.569383 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a9f95b6-ab91-4ce1-98ad-850426a2483e None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:15.571095 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-9a9f95b6-ab91-4ce1-98ad-850426a2483e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:15.584282 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a9f95b6-ab91-4ce1-98ad-850426a2483e None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:15.589543 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01ba3ffa-769d-48ae-8e96-7fe673d720d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:15.590174 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ba3ffa-769d-48ae-8e96-7fe673d720d0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:15.590443 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ba3ffa-769d-48ae-8e96-7fe673d720d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:15.590769 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ba3ffa-769d-48ae-8e96-7fe673d720d0 admin admin] PATH_INFO: `/v3/projects/a558aced0de64bae905abf66d5fa8736` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:15.591983 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01ba3ffa-769d-48ae-8e96-7fe673d720d0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a558aced0de64bae905abf66d5fa8736)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:15.593882 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01ba3ffa-769d-48ae-8e96-7fe673d720d0 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:15.657753 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-01ba3ffa-769d-48ae-8e96-7fe673d720d0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a558aced0de64bae905abf66d5fa8736', 'request_id': 'req-01ba3ffa-769d-48ae-8e96-7fe673d720d0'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:15.666676 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-01ba3ffa-769d-48ae-8e96-7fe673d720d0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a558aced0de64bae905abf66d5fa8736', 'request_id': 'req-01ba3ffa-769d-48ae-8e96-7fe673d720d0'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:15.672651 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-01ba3ffa-769d-48ae-8e96-7fe673d720d0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a558aced0de64bae905abf66d5fa8736', 'request_id': 'req-01ba3ffa-769d-48ae-8e96-7fe673d720d0'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:15.685808 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-01ba3ffa-769d-48ae-8e96-7fe673d720d0 admin admin] The token cache is being invalidate because project a558aced0de64bae905abf66d5fa8736 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87923) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:54:15.686131 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-01ba3ffa-769d-48ae-8e96-7fe673d720d0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:15.686726 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-01ba3ffa-769d-48ae-8e96-7fe673d720d0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:15.688340 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1095/2190] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:54:15 2022] DELETE /identity/v3/projects/a558aced0de64bae905abf66d5fa8736 => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:54:15.694827 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5192deb6-f342-48b0-bbce-b8c3176faa5a service neutron] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:15.706227 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5192deb6-f342-48b0-bbce-b8c3176faa5a service neutron] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:15.735245 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:15.766141 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a91631-e7f9-4cd2-9b05-7978c3a1416b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:15.766770 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a91631-e7f9-4cd2-9b05-7978c3a1416b admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:15.767079 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a91631-e7f9-4cd2-9b05-7978c3a1416b admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:15.767346 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a91631-e7f9-4cd2-9b05-7978c3a1416b admin admin] PATH_INFO: `/v3/users/0d9f8b30237642d789f1e66d05478d3e` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:15.768453 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3a91631-e7f9-4cd2-9b05-7978c3a1416b admin admin] RBAC: Authorizing `identity:delete_user(user_id=0d9f8b30237642d789f1e66d05478d3e)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:15.770354 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3a91631-e7f9-4cd2-9b05-7978c3a1416b admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:15.826166 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-f3a91631-e7f9-4cd2-9b05-7978c3a1416b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0d9f8b30237642d789f1e66d05478d3e'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:15.826825 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-f3a91631-e7f9-4cd2-9b05-7978c3a1416b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0d9f8b30237642d789f1e66d05478d3e'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:15.827431 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-f3a91631-e7f9-4cd2-9b05-7978c3a1416b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0d9f8b30237642d789f1e66d05478d3e'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:15.836472 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-f3a91631-e7f9-4cd2-9b05-7978c3a1416b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0d9f8b30237642d789f1e66d05478d3e'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:15.848453 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-f3a91631-e7f9-4cd2-9b05-7978c3a1416b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0d9f8b30237642d789f1e66d05478d3e'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:15.874420 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1096/2191] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:54:15 2022] DELETE /identity/v3/users/0d9f8b30237642d789f1e66d05478d3e => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:54:16.263883 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01ba3ffa-769d-48ae-8e96-7fe673d720d0 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:16.265586 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-01ba3ffa-769d-48ae-8e96-7fe673d720d0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:16.289321 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01ba3ffa-769d-48ae-8e96-7fe673d720d0 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:16.310577 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:16.354620 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2811e36-c280-4684-b3b3-c40bc525c4e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:16.355335 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2811e36-c280-4684-b3b3-c40bc525c4e8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:16.355620 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2811e36-c280-4684-b3b3-c40bc525c4e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:16.355897 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2811e36-c280-4684-b3b3-c40bc525c4e8 admin admin] PATH_INFO: `/v3/projects/480066f74364450eb0e137b28c77e9f9` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:16.357052 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2811e36-c280-4684-b3b3-c40bc525c4e8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=480066f74364450eb0e137b28c77e9f9)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:16.359006 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2811e36-c280-4684-b3b3-c40bc525c4e8 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:16.410721 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-d2811e36-c280-4684-b3b3-c40bc525c4e8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '480066f74364450eb0e137b28c77e9f9', 'request_id': 'req-d2811e36-c280-4684-b3b3-c40bc525c4e8'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:16.417604 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-d2811e36-c280-4684-b3b3-c40bc525c4e8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '480066f74364450eb0e137b28c77e9f9', 'request_id': 'req-d2811e36-c280-4684-b3b3-c40bc525c4e8'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:16.423960 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-d2811e36-c280-4684-b3b3-c40bc525c4e8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '480066f74364450eb0e137b28c77e9f9', 'request_id': 'req-d2811e36-c280-4684-b3b3-c40bc525c4e8'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:16.437358 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-d2811e36-c280-4684-b3b3-c40bc525c4e8 admin admin] The token cache is being invalidate because project 480066f74364450eb0e137b28c77e9f9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87923) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:54:16.437570 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-d2811e36-c280-4684-b3b3-c40bc525c4e8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:16.438207 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-d2811e36-c280-4684-b3b3-c40bc525c4e8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:16.439753 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1096/2192] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:54:16 2022] DELETE /identity/v3/projects/480066f74364450eb0e137b28c77e9f9 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:54:16.444697 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5e0ecd-2a27-4622-82dc-c4220c881b98 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:16.445183 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5e0ecd-2a27-4622-82dc-c4220c881b98 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:16.445333 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5e0ecd-2a27-4622-82dc-c4220c881b98 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:16.466489 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-7c5e0ecd-2a27-4622-82dc-c4220c881b98 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:16.467764 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-7c5e0ecd-2a27-4622-82dc-c4220c881b98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:16.518992 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2811e36-c280-4684-b3b3-c40bc525c4e8 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:16.520918 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-d2811e36-c280-4684-b3b3-c40bc525c4e8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:16.528401 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1097/2193] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:54:16.534428 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2811e36-c280-4684-b3b3-c40bc525c4e8 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:16.536064 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc0421e-91c5-4af3-8e56-8be2ae2d55cb None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:16.536427 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc0421e-91c5-4af3-8e56-8be2ae2d55cb None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:16.536779 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc0421e-91c5-4af3-8e56-8be2ae2d55cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:16.539545 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a0d4164-cca8-4269-9f54-bf8beb0183de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:16.540312 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0d4164-cca8-4269-9f54-bf8beb0183de admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:16.540752 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0d4164-cca8-4269-9f54-bf8beb0183de admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:16.540852 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0d4164-cca8-4269-9f54-bf8beb0183de admin admin] PATH_INFO: `/v3/users/90b14790bd90497c8d6c5f6705e0d44d` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:16.543499 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a0d4164-cca8-4269-9f54-bf8beb0183de admin admin] RBAC: Authorizing `identity:delete_user(user_id=90b14790bd90497c8d6c5f6705e0d44d)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:16.545687 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a0d4164-cca8-4269-9f54-bf8beb0183de admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:16.564928 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-9fc0421e-91c5-4af3-8e56-8be2ae2d55cb None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:16.566307 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-9fc0421e-91c5-4af3-8e56-8be2ae2d55cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:16.574398 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1098/2194] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:54:16.578693 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc0421e-91c5-4af3-8e56-8be2ae2d55cb None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:16.589566 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc0421e-91c5-4af3-8e56-8be2ae2d55cb None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:16.594713 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5299a1aa-dd3d-4011-838c-ee8d7ee16545 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:16.595361 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5299a1aa-dd3d-4011-838c-ee8d7ee16545 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:16.595633 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5299a1aa-dd3d-4011-838c-ee8d7ee16545 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:16.595899 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5299a1aa-dd3d-4011-838c-ee8d7ee16545 admin admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:16.596537 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5299a1aa-dd3d-4011-838c-ee8d7ee16545 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:54:16.596833 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5299a1aa-dd3d-4011-838c-ee8d7ee16545 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:16.598402 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5299a1aa-dd3d-4011-838c-ee8d7ee16545 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:16.602915 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-1a0d4164-cca8-4269-9f54-bf8beb0183de admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '90b14790bd90497c8d6c5f6705e0d44d'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:16.603587 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-1a0d4164-cca8-4269-9f54-bf8beb0183de admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '90b14790bd90497c8d6c5f6705e0d44d'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:16.604275 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-1a0d4164-cca8-4269-9f54-bf8beb0183de admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '90b14790bd90497c8d6c5f6705e0d44d'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:16.610156 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1099/2195] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 00:54:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:54:16.614318 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5299a1aa-dd3d-4011-838c-ee8d7ee16545 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:16.615005 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-1a0d4164-cca8-4269-9f54-bf8beb0183de admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '90b14790bd90497c8d6c5f6705e0d44d'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:16.615959 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-5299a1aa-dd3d-4011-838c-ee8d7ee16545 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:16.628383 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-1a0d4164-cca8-4269-9f54-bf8beb0183de admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '90b14790bd90497c8d6c5f6705e0d44d'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:16.629390 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5299a1aa-dd3d-4011-838c-ee8d7ee16545 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:16.634269 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a75daee8-4280-4dfb-b406-c8cc4d5db842 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:16.634946 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75daee8-4280-4dfb-b406-c8cc4d5db842 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:16.635232 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75daee8-4280-4dfb-b406-c8cc4d5db842 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:16.635553 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75daee8-4280-4dfb-b406-c8cc4d5db842 admin admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:16.636317 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a75daee8-4280-4dfb-b406-c8cc4d5db842 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:16.638375 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a75daee8-4280-4dfb-b406-c8cc4d5db842 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:16.642132 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1097/2196] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:54:16 2022] DELETE /identity/v3/users/90b14790bd90497c8d6c5f6705e0d44d => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:54:16.646629 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c2d558-0df1-4d6d-a66f-bf1334ac419d None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:16.646963 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c2d558-0df1-4d6d-a66f-bf1334ac419d None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:16.647237 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c2d558-0df1-4d6d-a66f-bf1334ac419d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:16.667292 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-76c2d558-0df1-4d6d-a66f-bf1334ac419d None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:16.668763 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-76c2d558-0df1-4d6d-a66f-bf1334ac419d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:16.668893 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1100/2197] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:54:16 2022] POST /identity/v3/projects => generated 383 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:54:16.672604 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a75daee8-4280-4dfb-b406-c8cc4d5db842 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:16.678223 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:16.687664 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:16.730221 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1098/2198] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:54:16.739488 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a75daee8-4280-4dfb-b406-c8cc4d5db842 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:16.739932 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c2d558-0df1-4d6d-a66f-bf1334ac419d None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:16.741405 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-76c2d558-0df1-4d6d-a66f-bf1334ac419d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:16.744156 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0cde956-e21f-4f0c-89e9-29e639d3806f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:16.745297 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0cde956-e21f-4f0c-89e9-29e639d3806f admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:16.745577 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0cde956-e21f-4f0c-89e9-29e639d3806f admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:16.745955 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0cde956-e21f-4f0c-89e9-29e639d3806f admin admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:16.746597 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0cde956-e21f-4f0c-89e9-29e639d3806f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:16.748716 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0cde956-e21f-4f0c-89e9-29e639d3806f admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:16.773473 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1101/2199] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:54:16 2022] POST /identity/v3/users => generated 292 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:54:16.774601 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0cde956-e21f-4f0c-89e9-29e639d3806f admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:16.786050 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0cde956-e21f-4f0c-89e9-29e639d3806f admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:16.790777 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c2679e-b132-4432-bea3-2a9e44a8ea04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:16.791385 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c2679e-b132-4432-bea3-2a9e44a8ea04 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:16.791730 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c2679e-b132-4432-bea3-2a9e44a8ea04 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:16.792003 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c2679e-b132-4432-bea3-2a9e44a8ea04 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:16.792642 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30c2679e-b132-4432-bea3-2a9e44a8ea04 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:16.794426 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30c2679e-b132-4432-bea3-2a9e44a8ea04 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:16.802305 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c2d558-0df1-4d6d-a66f-bf1334ac419d None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:16.805984 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1102/2200] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:54:16 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:54:16.809928 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fceb9219-a9a8-4521-acf7-3645a84f9022 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:16.809928 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fceb9219-a9a8-4521-acf7-3645a84f9022 service neutron] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:16.809928 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fceb9219-a9a8-4521-acf7-3645a84f9022 service neutron] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:16.809928 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fceb9219-a9a8-4521-acf7-3645a84f9022 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:16.810849 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c2679e-b132-4432-bea3-2a9e44a8ea04 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:16.821220 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fceb9219-a9a8-4521-acf7-3645a84f9022 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:16.822029 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c2679e-b132-4432-bea3-2a9e44a8ea04 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:16.822860 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fceb9219-a9a8-4521-acf7-3645a84f9022 service neutron] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:16.827608 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04959ddf-5192-4fad-9ef6-4f7e14dc68ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:16.828206 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04959ddf-5192-4fad-9ef6-4f7e14dc68ae admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:16.828487 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04959ddf-5192-4fad-9ef6-4f7e14dc68ae admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:16.828820 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04959ddf-5192-4fad-9ef6-4f7e14dc68ae admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:16.829463 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04959ddf-5192-4fad-9ef6-4f7e14dc68ae admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:16.831073 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04959ddf-5192-4fad-9ef6-4f7e14dc68ae admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:16.833966 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1099/2201] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:54:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 11 00:54:16.842043 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1103/2202] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:54:16 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 11 00:54:16.846235 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fceb9219-a9a8-4521-acf7-3645a84f9022 service neutron] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:16.856654 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fceb9219-a9a8-4521-acf7-3645a84f9022 service neutron] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:16.861479 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e724656d-e174-4d33-b53e-6cabfe29f4f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:16.862128 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e724656d-e174-4d33-b53e-6cabfe29f4f7 admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:16.862409 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e724656d-e174-4d33-b53e-6cabfe29f4f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:16.862686 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e724656d-e174-4d33-b53e-6cabfe29f4f7 admin admin] PATH_INFO: `/v3/projects/846640420cec4277a5cb7838d923bf95/users/b5e7fe0872704aa69520922889c10dd4/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:16.878691 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e724656d-e174-4d33-b53e-6cabfe29f4f7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=846640420cec4277a5cb7838d923bf95, user_id=b5e7fe0872704aa69520922889c10dd4, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:16.881498 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e724656d-e174-4d33-b53e-6cabfe29f4f7 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:16.894710 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1100/2203] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:54:16 2022] PUT /identity/v3/projects/846640420cec4277a5cb7838d923bf95/users/b5e7fe0872704aa69520922889c10dd4/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:54:16.905086 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76cd629e-28b6-4078-a9ff-5cd14082f3cd None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:16.905406 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76cd629e-28b6-4078-a9ff-5cd14082f3cd None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:16.905711 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76cd629e-28b6-4078-a9ff-5cd14082f3cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:16.938319 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-76cd629e-28b6-4078-a9ff-5cd14082f3cd None None] MFA Rules not processed for user `b5e7fe0872704aa69520922889c10dd4`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:16.940152 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-76cd629e-28b6-4078-a9ff-5cd14082f3cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:17.013551 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1104/2204] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:16 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:54:17.025370 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e724656d-e174-4d33-b53e-6cabfe29f4f7 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:17.027718 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-e724656d-e174-4d33-b53e-6cabfe29f4f7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:17.090713 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e724656d-e174-4d33-b53e-6cabfe29f4f7 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:17.106228 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d8d5bae-6cbd-4bc3-9b57-5134f7e9dd63 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:17.106924 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8d5bae-6cbd-4bc3-9b57-5134f7e9dd63 service cinder] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:17.107211 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8d5bae-6cbd-4bc3-9b57-5134f7e9dd63 service cinder] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:17.107482 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8d5bae-6cbd-4bc3-9b57-5134f7e9dd63 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:17.119488 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d8d5bae-6cbd-4bc3-9b57-5134f7e9dd63 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:17.122065 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d8d5bae-6cbd-4bc3-9b57-5134f7e9dd63 service cinder] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:17.133677 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76cd629e-28b6-4078-a9ff-5cd14082f3cd None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:17.135679 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-76cd629e-28b6-4078-a9ff-5cd14082f3cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:17.136312 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1101/2205] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:54:17 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:54:17.166103 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d8d5bae-6cbd-4bc3-9b57-5134f7e9dd63 service cinder] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:17.221750 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76cd629e-28b6-4078-a9ff-5cd14082f3cd None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:17.223878 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:17.227348 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7aadab1-11b7-47d8-88a3-455f3d7bae55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:17.227992 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7aadab1-11b7-47d8-88a3-455f3d7bae55 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:17.228264 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7aadab1-11b7-47d8-88a3-455f3d7bae55 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:17.228531 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7aadab1-11b7-47d8-88a3-455f3d7bae55 admin admin] PATH_INFO: `/v3/projects/ff59703649fa43269925aa9065689849` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:17.230302 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7aadab1-11b7-47d8-88a3-455f3d7bae55 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ff59703649fa43269925aa9065689849)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:17.233047 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7aadab1-11b7-47d8-88a3-455f3d7bae55 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:17.235620 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:17.242609 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d8d5bae-6cbd-4bc3-9b57-5134f7e9dd63 service cinder] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:17.249512 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16e376cc-9126-4b8b-af74-777df82a0b9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:17.250159 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e376cc-9126-4b8b-af74-777df82a0b9a admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:17.250439 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e376cc-9126-4b8b-af74-777df82a0b9a admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:17.250713 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e376cc-9126-4b8b-af74-777df82a0b9a admin admin] PATH_INFO: `/v3/users/b5e7fe0872704aa69520922889c10dd4` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:17.251913 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16e376cc-9126-4b8b-af74-777df82a0b9a admin admin] RBAC: Authorizing `identity:delete_user(user_id=b5e7fe0872704aa69520922889c10dd4)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:17.254250 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16e376cc-9126-4b8b-af74-777df82a0b9a admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:17.297013 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-a7aadab1-11b7-47d8-88a3-455f3d7bae55 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ff59703649fa43269925aa9065689849', 'request_id': 'req-a7aadab1-11b7-47d8-88a3-455f3d7bae55'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:17.304639 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-a7aadab1-11b7-47d8-88a3-455f3d7bae55 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ff59703649fa43269925aa9065689849', 'request_id': 'req-a7aadab1-11b7-47d8-88a3-455f3d7bae55'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:17.312417 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-a7aadab1-11b7-47d8-88a3-455f3d7bae55 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ff59703649fa43269925aa9065689849', 'request_id': 'req-a7aadab1-11b7-47d8-88a3-455f3d7bae55'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:17.317201 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-16e376cc-9126-4b8b-af74-777df82a0b9a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b5e7fe0872704aa69520922889c10dd4'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:17.317868 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-16e376cc-9126-4b8b-af74-777df82a0b9a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b5e7fe0872704aa69520922889c10dd4'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:17.318512 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-16e376cc-9126-4b8b-af74-777df82a0b9a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b5e7fe0872704aa69520922889c10dd4'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:17.324590 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-a7aadab1-11b7-47d8-88a3-455f3d7bae55 admin admin] The token cache is being invalidate because project ff59703649fa43269925aa9065689849 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87924) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:54:17.324876 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-a7aadab1-11b7-47d8-88a3-455f3d7bae55 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:17.325755 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-a7aadab1-11b7-47d8-88a3-455f3d7bae55 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:17.326989 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-16e376cc-9126-4b8b-af74-777df82a0b9a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b5e7fe0872704aa69520922889c10dd4'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:17.327302 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1105/2206] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:54:17 2022] DELETE /identity/v3/projects/ff59703649fa43269925aa9065689849 => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:54:17.331240 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7aadab1-11b7-47d8-88a3-455f3d7bae55 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:17.332643 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-a7aadab1-11b7-47d8-88a3-455f3d7bae55 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:17.340344 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-16e376cc-9126-4b8b-af74-777df82a0b9a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b5e7fe0872704aa69520922889c10dd4'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:17.354143 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1102/2207] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:54:17 2022] DELETE /identity/v3/users/b5e7fe0872704aa69520922889c10dd4 => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:54:17.358704 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f18afec-014c-49b4-ad22-d959c477c65f None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:17.359026 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f18afec-014c-49b4-ad22-d959c477c65f None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:17.359308 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f18afec-014c-49b4-ad22-d959c477c65f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:17.378857 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-6f18afec-014c-49b4-ad22-d959c477c65f None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:17.380056 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-6f18afec-014c-49b4-ad22-d959c477c65f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:17.398712 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7aadab1-11b7-47d8-88a3-455f3d7bae55 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:17.440645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1103/2208] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:54:17.451421 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f18afec-014c-49b4-ad22-d959c477c65f None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:17.452846 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-6f18afec-014c-49b4-ad22-d959c477c65f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:17.461983 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7167158f-9d04-4b1f-817e-07f6cf273e75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:17.462620 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7167158f-9d04-4b1f-817e-07f6cf273e75 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:17.462948 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7167158f-9d04-4b1f-817e-07f6cf273e75 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:17.463224 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7167158f-9d04-4b1f-817e-07f6cf273e75 admin admin] PATH_INFO: `/v3/users/e5d00c46be13447395f6f09015de93d1` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:17.464423 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7167158f-9d04-4b1f-817e-07f6cf273e75 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e5d00c46be13447395f6f09015de93d1)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:17.466449 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7167158f-9d04-4b1f-817e-07f6cf273e75 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:17.507956 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:17.511086 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f18afec-014c-49b4-ad22-d959c477c65f None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:17.513675 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-7167158f-9d04-4b1f-817e-07f6cf273e75 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e5d00c46be13447395f6f09015de93d1'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:17.514423 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-7167158f-9d04-4b1f-817e-07f6cf273e75 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e5d00c46be13447395f6f09015de93d1'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:17.515200 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcebb355-ab14-4224-b3b8-9e5e9d6b128e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:17.515576 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-7167158f-9d04-4b1f-817e-07f6cf273e75 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e5d00c46be13447395f6f09015de93d1'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:17.515843 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcebb355-ab14-4224-b3b8-9e5e9d6b128e service neutron] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:17.516156 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcebb355-ab14-4224-b3b8-9e5e9d6b128e service neutron] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:17.516505 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcebb355-ab14-4224-b3b8-9e5e9d6b128e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:17.518580 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-fcebb355-ab14-4224-b3b8-9e5e9d6b128e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:17.523045 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-7167158f-9d04-4b1f-817e-07f6cf273e75 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e5d00c46be13447395f6f09015de93d1'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:17.531210 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcebb355-ab14-4224-b3b8-9e5e9d6b128e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:17.532657 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcebb355-ab14-4224-b3b8-9e5e9d6b128e service neutron] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:17.535896 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-7167158f-9d04-4b1f-817e-07f6cf273e75 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e5d00c46be13447395f6f09015de93d1'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:17.542903 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1104/2209] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:54:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:54:17.552243 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1106/2210] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:54:17 2022] DELETE /identity/v3/users/e5d00c46be13447395f6f09015de93d1 => generated 0 bytes in 221 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:54:17.902305 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcebb355-ab14-4224-b3b8-9e5e9d6b128e service neutron] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:17.903611 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-fcebb355-ab14-4224-b3b8-9e5e9d6b128e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:17.909230 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7167158f-9d04-4b1f-817e-07f6cf273e75 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:17.911140 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-7167158f-9d04-4b1f-817e-07f6cf273e75 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:17.918508 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcebb355-ab14-4224-b3b8-9e5e9d6b128e service neutron] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:17.926862 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:17.987880 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327c5741-c21a-4de9-b977-21c911513669 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:17.988550 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327c5741-c21a-4de9-b977-21c911513669 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:17.988826 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327c5741-c21a-4de9-b977-21c911513669 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:17.989101 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327c5741-c21a-4de9-b977-21c911513669 admin admin] PATH_INFO: `/v3/projects/846640420cec4277a5cb7838d923bf95` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:17.990236 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-327c5741-c21a-4de9-b977-21c911513669 admin admin] RBAC: Authorizing `identity:delete_project(project_id=846640420cec4277a5cb7838d923bf95)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:17.992198 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-327c5741-c21a-4de9-b977-21c911513669 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:17.994416 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7167158f-9d04-4b1f-817e-07f6cf273e75 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:17.999129 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9901e13-446b-4846-a205-423aeba98c97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:17.999759 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9901e13-446b-4846-a205-423aeba98c97 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:18.000038 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9901e13-446b-4846-a205-423aeba98c97 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:18.000308 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9901e13-446b-4846-a205-423aeba98c97 admin admin] PATH_INFO: `/v3/projects/569a9686a0244e2781f1e9beca6bc74a` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:18.001437 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9901e13-446b-4846-a205-423aeba98c97 admin admin] RBAC: Authorizing `identity:delete_project(project_id=569a9686a0244e2781f1e9beca6bc74a)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:18.003385 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9901e13-446b-4846-a205-423aeba98c97 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:18.040767 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-327c5741-c21a-4de9-b977-21c911513669 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '846640420cec4277a5cb7838d923bf95', 'request_id': 'req-327c5741-c21a-4de9-b977-21c911513669'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:18.047737 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-327c5741-c21a-4de9-b977-21c911513669 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '846640420cec4277a5cb7838d923bf95', 'request_id': 'req-327c5741-c21a-4de9-b977-21c911513669'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:18.053843 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-327c5741-c21a-4de9-b977-21c911513669 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '846640420cec4277a5cb7838d923bf95', 'request_id': 'req-327c5741-c21a-4de9-b977-21c911513669'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:18.057264 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-f9901e13-446b-4846-a205-423aeba98c97 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '569a9686a0244e2781f1e9beca6bc74a', 'request_id': 'req-f9901e13-446b-4846-a205-423aeba98c97'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:18.064429 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-f9901e13-446b-4846-a205-423aeba98c97 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '569a9686a0244e2781f1e9beca6bc74a', 'request_id': 'req-f9901e13-446b-4846-a205-423aeba98c97'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:18.067317 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-327c5741-c21a-4de9-b977-21c911513669 admin admin] The token cache is being invalidate because project 846640420cec4277a5cb7838d923bf95 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87923) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:54:18.067603 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-327c5741-c21a-4de9-b977-21c911513669 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:18.068283 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-327c5741-c21a-4de9-b977-21c911513669 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:18.069817 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1105/2211] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:54:17 2022] DELETE /identity/v3/projects/846640420cec4277a5cb7838d923bf95 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:54:18.073720 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-f9901e13-446b-4846-a205-423aeba98c97 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '569a9686a0244e2781f1e9beca6bc74a', 'request_id': 'req-f9901e13-446b-4846-a205-423aeba98c97'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:18.074628 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f737fd11-745c-4b92-bdad-c0b96c9fdd07 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:18.075008 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f737fd11-745c-4b92-bdad-c0b96c9fdd07 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:18.075288 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f737fd11-745c-4b92-bdad-c0b96c9fdd07 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:18.094854 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-f9901e13-446b-4846-a205-423aeba98c97 admin admin] The token cache is being invalidate because project 569a9686a0244e2781f1e9beca6bc74a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87924) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:54:18.095171 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-f9901e13-446b-4846-a205-423aeba98c97 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:18.095832 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-f9901e13-446b-4846-a205-423aeba98c97 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:18.097510 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1107/2212] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:54:17 2022] DELETE /identity/v3/projects/569a9686a0244e2781f1e9beca6bc74a => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:54:18.102426 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9901e13-446b-4846-a205-423aeba98c97 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:18.103857 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-f9901e13-446b-4846-a205-423aeba98c97 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:18.108633 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-f737fd11-745c-4b92-bdad-c0b96c9fdd07 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:18.111650 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-f737fd11-745c-4b92-bdad-c0b96c9fdd07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:18.156984 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=87,keystone:DELETE=11,keystone:INSERT=2 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:18.178152 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9901e13-446b-4846-a205-423aeba98c97 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:18.178942 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1106/2213] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:54:18.188456 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3dbb681-3259-449f-9c05-978cc248f7d8 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:18.188886 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3dbb681-3259-449f-9c05-978cc248f7d8 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:18.189163 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3dbb681-3259-449f-9c05-978cc248f7d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:18.189685 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f29e6c-f5de-45e8-a81a-60becc1e5953 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:18.190470 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f29e6c-f5de-45e8-a81a-60becc1e5953 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:18.190908 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f29e6c-f5de-45e8-a81a-60becc1e5953 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:18.191284 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f29e6c-f5de-45e8-a81a-60becc1e5953 admin admin] PATH_INFO: `/v3/users/6070a5fec40348a589c73a7fc8791a52` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:18.192525 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1f29e6c-f5de-45e8-a81a-60becc1e5953 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6070a5fec40348a589c73a7fc8791a52)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:18.194649 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1f29e6c-f5de-45e8-a81a-60becc1e5953 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:18.208590 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-c3dbb681-3259-449f-9c05-978cc248f7d8 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:18.209788 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-c3dbb681-3259-449f-9c05-978cc248f7d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:18.218401 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1107/2214] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:54:18.222936 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3dbb681-3259-449f-9c05-978cc248f7d8 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:18.234152 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3dbb681-3259-449f-9c05-978cc248f7d8 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:18.238877 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b86c45a9-1005-43af-9af5-9609b423bcbd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:18.239468 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86c45a9-1005-43af-9af5-9609b423bcbd admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:18.239747 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86c45a9-1005-43af-9af5-9609b423bcbd admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:18.240059 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86c45a9-1005-43af-9af5-9609b423bcbd admin admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:18.240710 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b86c45a9-1005-43af-9af5-9609b423bcbd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:54:18.241018 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b86c45a9-1005-43af-9af5-9609b423bcbd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:18.242605 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b86c45a9-1005-43af-9af5-9609b423bcbd admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:18.243882 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-c1f29e6c-f5de-45e8-a81a-60becc1e5953 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6070a5fec40348a589c73a7fc8791a52'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:18.244559 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-c1f29e6c-f5de-45e8-a81a-60becc1e5953 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6070a5fec40348a589c73a7fc8791a52'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:18.245240 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-c1f29e6c-f5de-45e8-a81a-60becc1e5953 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6070a5fec40348a589c73a7fc8791a52'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:18.253177 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-c1f29e6c-f5de-45e8-a81a-60becc1e5953 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6070a5fec40348a589c73a7fc8791a52'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:18.255228 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1108/2215] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 00:54:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:54:18.259116 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b86c45a9-1005-43af-9af5-9609b423bcbd admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:18.260819 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-b86c45a9-1005-43af-9af5-9609b423bcbd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:18.266438 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-c1f29e6c-f5de-45e8-a81a-60becc1e5953 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6070a5fec40348a589c73a7fc8791a52'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:18.273148 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b86c45a9-1005-43af-9af5-9609b423bcbd admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:18.278698 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-352e8e0a-9b13-4dac-87dd-9fe3570d40fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:18.279320 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-352e8e0a-9b13-4dac-87dd-9fe3570d40fa admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:18.279597 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-352e8e0a-9b13-4dac-87dd-9fe3570d40fa admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:18.279865 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-352e8e0a-9b13-4dac-87dd-9fe3570d40fa admin admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:18.280107 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1108/2216] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:54:18 2022] DELETE /identity/v3/users/6070a5fec40348a589c73a7fc8791a52 => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:54:18.280487 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-352e8e0a-9b13-4dac-87dd-9fe3570d40fa admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:18.282409 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-352e8e0a-9b13-4dac-87dd-9fe3570d40fa admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:18.304541 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:18.318498 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1109/2217] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:54:18 2022] POST /identity/v3/projects => generated 383 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:54:18.319131 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f29e6c-f5de-45e8-a81a-60becc1e5953 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:18.406771 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f29e6c-f5de-45e8-a81a-60becc1e5953 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:18.491545 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4703a371-4271-4712-b268-6f8f138d59c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:18.492246 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4703a371-4271-4712-b268-6f8f138d59c0 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:18.492456 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4703a371-4271-4712-b268-6f8f138d59c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:18.492732 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4703a371-4271-4712-b268-6f8f138d59c0 admin admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:18.493365 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4703a371-4271-4712-b268-6f8f138d59c0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:18.507004 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4703a371-4271-4712-b268-6f8f138d59c0 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:18.522867 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1109/2218] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:54:18 2022] POST /identity/v3/users => generated 292 bytes in 202 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:54:18.524083 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-352e8e0a-9b13-4dac-87dd-9fe3570d40fa admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:18.535154 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-352e8e0a-9b13-4dac-87dd-9fe3570d40fa admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:18.540139 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f53dc0b-351f-4232-b371-6a66a7e31f6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:18.540835 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f53dc0b-351f-4232-b371-6a66a7e31f6f admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:18.541169 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f53dc0b-351f-4232-b371-6a66a7e31f6f admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:18.541460 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f53dc0b-351f-4232-b371-6a66a7e31f6f admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:18.542877 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f53dc0b-351f-4232-b371-6a66a7e31f6f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:18.544463 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f53dc0b-351f-4232-b371-6a66a7e31f6f admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:18.559298 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1110/2219] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:54:18 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 11 00:54:18.560549 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4703a371-4271-4712-b268-6f8f138d59c0 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:18.567039 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:18.571499 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4703a371-4271-4712-b268-6f8f138d59c0 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:18.577922 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313513ea-5e45-4ac9-a0ce-f1033606f4d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:18.578665 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313513ea-5e45-4ac9-a0ce-f1033606f4d2 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:18.579095 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313513ea-5e45-4ac9-a0ce-f1033606f4d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:18.579475 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313513ea-5e45-4ac9-a0ce-f1033606f4d2 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:18.580294 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-313513ea-5e45-4ac9-a0ce-f1033606f4d2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:18.582132 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-313513ea-5e45-4ac9-a0ce-f1033606f4d2 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:18.600376 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f53dc0b-351f-4232-b371-6a66a7e31f6f admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:18.602285 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-3f53dc0b-351f-4232-b371-6a66a7e31f6f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:18.614835 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1110/2220] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:54:18 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:54:18.620499 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313513ea-5e45-4ac9-a0ce-f1033606f4d2 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:18.624019 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f53dc0b-351f-4232-b371-6a66a7e31f6f admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:18.631148 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a09cbc4-67a5-4adf-ad6c-fd3e30c18be5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:18.631849 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a09cbc4-67a5-4adf-ad6c-fd3e30c18be5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:18.632013 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a09cbc4-67a5-4adf-ad6c-fd3e30c18be5 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:18.632130 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313513ea-5e45-4ac9-a0ce-f1033606f4d2 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:18.632281 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a09cbc4-67a5-4adf-ad6c-fd3e30c18be5 admin admin] PATH_INFO: `/v3/projects/4c31e6c9ef2c432386ba8e2a0a80b408` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:18.633533 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a09cbc4-67a5-4adf-ad6c-fd3e30c18be5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4c31e6c9ef2c432386ba8e2a0a80b408)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:18.635533 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a09cbc4-67a5-4adf-ad6c-fd3e30c18be5 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:18.637280 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd1faf0a-c987-47d2-9f1b-1e310c121f61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:18.637938 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd1faf0a-c987-47d2-9f1b-1e310c121f61 admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:18.638219 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd1faf0a-c987-47d2-9f1b-1e310c121f61 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:18.638493 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd1faf0a-c987-47d2-9f1b-1e310c121f61 admin admin] PATH_INFO: `/v3/projects/15d1340dbc59490ca4d9c56802b9cef6/users/0cb02c3d58444627913b6462fd1090d2/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:18.655432 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd1faf0a-c987-47d2-9f1b-1e310c121f61 admin admin] RBAC: Authorizing `identity:create_grant(project_id=15d1340dbc59490ca4d9c56802b9cef6, user_id=0cb02c3d58444627913b6462fd1090d2, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:18.658541 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd1faf0a-c987-47d2-9f1b-1e310c121f61 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:18.673494 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1111/2221] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:54:18 2022] PUT /identity/v3/projects/15d1340dbc59490ca4d9c56802b9cef6/users/0cb02c3d58444627913b6462fd1090d2/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:54:18.684720 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99354826-a7b3-4095-b13c-f03df443b6be None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:18.685038 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99354826-a7b3-4095-b13c-f03df443b6be None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:18.685317 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99354826-a7b3-4095-b13c-f03df443b6be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:18.687187 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-1a09cbc4-67a5-4adf-ad6c-fd3e30c18be5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4c31e6c9ef2c432386ba8e2a0a80b408', 'request_id': 'req-1a09cbc4-67a5-4adf-ad6c-fd3e30c18be5'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:18.697051 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-1a09cbc4-67a5-4adf-ad6c-fd3e30c18be5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4c31e6c9ef2c432386ba8e2a0a80b408', 'request_id': 'req-1a09cbc4-67a5-4adf-ad6c-fd3e30c18be5'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:18.704267 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-1a09cbc4-67a5-4adf-ad6c-fd3e30c18be5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4c31e6c9ef2c432386ba8e2a0a80b408', 'request_id': 'req-1a09cbc4-67a5-4adf-ad6c-fd3e30c18be5'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:18.717167 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-1a09cbc4-67a5-4adf-ad6c-fd3e30c18be5 admin admin] The token cache is being invalidate because project 4c31e6c9ef2c432386ba8e2a0a80b408 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87923) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:54:18.717444 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-1a09cbc4-67a5-4adf-ad6c-fd3e30c18be5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:18.718183 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-1a09cbc4-67a5-4adf-ad6c-fd3e30c18be5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:18.719470 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-99354826-a7b3-4095-b13c-f03df443b6be None None] MFA Rules not processed for user `0cb02c3d58444627913b6462fd1090d2`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:18.719755 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1111/2222] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:54:18 2022] DELETE /identity/v3/projects/4c31e6c9ef2c432386ba8e2a0a80b408 => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:54:18.720705 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-99354826-a7b3-4095-b13c-f03df443b6be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:18.725510 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226e42dd-9377-4f9e-a8fa-01e6d06a2062 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:18.725788 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226e42dd-9377-4f9e-a8fa-01e6d06a2062 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:18.726052 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226e42dd-9377-4f9e-a8fa-01e6d06a2062 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:18.746357 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-226e42dd-9377-4f9e-a8fa-01e6d06a2062 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:18.747726 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-226e42dd-9377-4f9e-a8fa-01e6d06a2062 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:18.792803 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1112/2223] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:18 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:54:18.797276 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99354826-a7b3-4095-b13c-f03df443b6be None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:18.798588 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-99354826-a7b3-4095-b13c-f03df443b6be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:18.813059 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1112/2224] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:54:18.823013 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f897377-c15a-41ba-9278-a599cce04dfd None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:18.823339 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f897377-c15a-41ba-9278-a599cce04dfd None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:18.823612 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f897377-c15a-41ba-9278-a599cce04dfd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:18.846609 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-9f897377-c15a-41ba-9278-a599cce04dfd None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:18.847833 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-9f897377-c15a-41ba-9278-a599cce04dfd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:18.856428 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1113/2225] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:54:18.860871 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f897377-c15a-41ba-9278-a599cce04dfd None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:18.872369 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f897377-c15a-41ba-9278-a599cce04dfd None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:18.873713 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99354826-a7b3-4095-b13c-f03df443b6be None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:18.877327 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1865a03c-05f0-4037-8111-582cca3262b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:18.877929 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1865a03c-05f0-4037-8111-582cca3262b0 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:18.878204 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1865a03c-05f0-4037-8111-582cca3262b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:18.878474 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1865a03c-05f0-4037-8111-582cca3262b0 admin admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:18.879165 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1865a03c-05f0-4037-8111-582cca3262b0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:54:18.879263 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5bbc4dd-7ecb-44d4-ac68-94dc6af51d06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:18.879470 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1865a03c-05f0-4037-8111-582cca3262b0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:18.880028 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5bbc4dd-7ecb-44d4-ac68-94dc6af51d06 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:18.880429 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5bbc4dd-7ecb-44d4-ac68-94dc6af51d06 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:18.880899 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5bbc4dd-7ecb-44d4-ac68-94dc6af51d06 admin admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:18.881074 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1865a03c-05f0-4037-8111-582cca3262b0 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:18.881539 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5bbc4dd-7ecb-44d4-ac68-94dc6af51d06 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:18.883522 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5bbc4dd-7ecb-44d4-ac68-94dc6af51d06 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:18.893629 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1114/2226] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 00:54:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:54:18.898076 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1865a03c-05f0-4037-8111-582cca3262b0 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:18.909243 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1865a03c-05f0-4037-8111-582cca3262b0 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:18.912351 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1113/2227] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:54:18 2022] POST /identity/v3/projects => generated 385 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:54:18.916712 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5bbc4dd-7ecb-44d4-ac68-94dc6af51d06 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:18.917362 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb93058c-fc41-4268-8035-7cbfd9074608 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:18.917972 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb93058c-fc41-4268-8035-7cbfd9074608 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:18.918255 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb93058c-fc41-4268-8035-7cbfd9074608 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:18.918529 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb93058c-fc41-4268-8035-7cbfd9074608 admin admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:18.919253 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb93058c-fc41-4268-8035-7cbfd9074608 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:18.921161 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb93058c-fc41-4268-8035-7cbfd9074608 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:18.956945 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1115/2228] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:54:18 2022] POST /identity/v3/projects => generated 387 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:54:18.958353 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb93058c-fc41-4268-8035-7cbfd9074608 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:18.970869 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:18.989266 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5bbc4dd-7ecb-44d4-ac68-94dc6af51d06 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:18.990684 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:19.037009 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb93058c-fc41-4268-8035-7cbfd9074608 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:19.041757 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2435a37a-3282-415d-959a-16fe6f94f330 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:19.042553 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2435a37a-3282-415d-959a-16fe6f94f330 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:19.042865 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2435a37a-3282-415d-959a-16fe6f94f330 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:19.043138 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2435a37a-3282-415d-959a-16fe6f94f330 admin admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:19.043768 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2435a37a-3282-415d-959a-16fe6f94f330 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:19.045678 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2435a37a-3282-415d-959a-16fe6f94f330 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:19.056991 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3314e38e-1e95-4611-aa22-724eeb267d42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:19.058215 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3314e38e-1e95-4611-aa22-724eeb267d42 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:19.058465 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3314e38e-1e95-4611-aa22-724eeb267d42 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:19.058892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3314e38e-1e95-4611-aa22-724eeb267d42 admin admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:19.059538 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3314e38e-1e95-4611-aa22-724eeb267d42 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:19.061744 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3314e38e-1e95-4611-aa22-724eeb267d42 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:19.070243 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1116/2229] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:54:18 2022] POST /identity/v3/users => generated 294 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:54:19.071178 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2435a37a-3282-415d-959a-16fe6f94f330 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:19.087708 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2435a37a-3282-415d-959a-16fe6f94f330 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:19.092547 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1114/2230] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:54:18 2022] POST /identity/v3/users => generated 293 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:54:19.092675 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f808789a-b429-49ac-a9df-64a8ca777918 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:19.093206 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f808789a-b429-49ac-a9df-64a8ca777918 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:19.093482 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f808789a-b429-49ac-a9df-64a8ca777918 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:19.093752 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f808789a-b429-49ac-a9df-64a8ca777918 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:19.094399 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f808789a-b429-49ac-a9df-64a8ca777918 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:19.096070 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f808789a-b429-49ac-a9df-64a8ca777918 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:19.097020 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3314e38e-1e95-4611-aa22-724eeb267d42 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:19.107522 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1117/2231] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:54:19 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 11 00:54:19.108212 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3314e38e-1e95-4611-aa22-724eeb267d42 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:19.111295 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f808789a-b429-49ac-a9df-64a8ca777918 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:19.113093 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a023a395-445d-44e8-a818-3137ded479ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:19.113695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a023a395-445d-44e8-a818-3137ded479ae admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:19.114347 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a023a395-445d-44e8-a818-3137ded479ae admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:19.114347 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a023a395-445d-44e8-a818-3137ded479ae admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:19.114895 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a023a395-445d-44e8-a818-3137ded479ae admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:19.116478 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a023a395-445d-44e8-a818-3137ded479ae admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:19.123208 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f808789a-b429-49ac-a9df-64a8ca777918 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:19.127425 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884ef316-c51b-4679-97bf-b730c9345ef9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:19.127594 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1115/2232] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:54:19 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 11 00:54:19.128026 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884ef316-c51b-4679-97bf-b730c9345ef9 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:19.128307 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884ef316-c51b-4679-97bf-b730c9345ef9 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:19.128573 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884ef316-c51b-4679-97bf-b730c9345ef9 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:19.129218 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-884ef316-c51b-4679-97bf-b730c9345ef9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:19.130917 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-884ef316-c51b-4679-97bf-b730c9345ef9 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:19.131304 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a023a395-445d-44e8-a818-3137ded479ae admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:19.145368 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1118/2233] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:54:19 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 11 00:54:19.145507 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a023a395-445d-44e8-a818-3137ded479ae admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:19.152884 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884ef316-c51b-4679-97bf-b730c9345ef9 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:19.156097 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f51ff68e-fd92-4c96-bca6-5e6bfa77d1b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:19.157101 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f51ff68e-fd92-4c96-bca6-5e6bfa77d1b3 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:19.157101 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f51ff68e-fd92-4c96-bca6-5e6bfa77d1b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:19.157540 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f51ff68e-fd92-4c96-bca6-5e6bfa77d1b3 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:19.158293 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f51ff68e-fd92-4c96-bca6-5e6bfa77d1b3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:19.160547 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f51ff68e-fd92-4c96-bca6-5e6bfa77d1b3 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:19.163433 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884ef316-c51b-4679-97bf-b730c9345ef9 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:19.168907 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df8bfe55-cbbe-47eb-95e2-d6f23b12d12a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:19.169594 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df8bfe55-cbbe-47eb-95e2-d6f23b12d12a admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:19.169865 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df8bfe55-cbbe-47eb-95e2-d6f23b12d12a admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:19.170111 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df8bfe55-cbbe-47eb-95e2-d6f23b12d12a admin admin] PATH_INFO: `/v3/projects/b78cf66aead646ff894292a407f7ca73/users/c13d417208d945c983c2073e84f507f2/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:19.173561 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1116/2234] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:54:19 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:54:19.177416 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f51ff68e-fd92-4c96-bca6-5e6bfa77d1b3 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:19.188013 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f51ff68e-fd92-4c96-bca6-5e6bfa77d1b3 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:19.188962 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df8bfe55-cbbe-47eb-95e2-d6f23b12d12a admin admin] RBAC: Authorizing `identity:create_grant(project_id=b78cf66aead646ff894292a407f7ca73, user_id=c13d417208d945c983c2073e84f507f2, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:19.191749 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df8bfe55-cbbe-47eb-95e2-d6f23b12d12a admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:19.192674 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce81f631-dcd2-4e7a-b33b-7128e03505e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:19.193363 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce81f631-dcd2-4e7a-b33b-7128e03505e2 admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:19.193642 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce81f631-dcd2-4e7a-b33b-7128e03505e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:19.193911 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce81f631-dcd2-4e7a-b33b-7128e03505e2 admin admin] PATH_INFO: `/v3/projects/dca34de5347f4f1d8ad840e8c4f81af1/users/0f2e48fce4844476bca569acef4a4a62/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:19.209824 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce81f631-dcd2-4e7a-b33b-7128e03505e2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dca34de5347f4f1d8ad840e8c4f81af1, user_id=0f2e48fce4844476bca569acef4a4a62, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:19.212761 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce81f631-dcd2-4e7a-b33b-7128e03505e2 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:19.216605 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1119/2235] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:54:19 2022] PUT /identity/v3/projects/b78cf66aead646ff894292a407f7ca73/users/c13d417208d945c983c2073e84f507f2/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:54:19.217494 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df8bfe55-cbbe-47eb-95e2-d6f23b12d12a admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:19.219480 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-df8bfe55-cbbe-47eb-95e2-d6f23b12d12a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:19.240484 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1117/2236] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:54:19 2022] PUT /identity/v3/projects/dca34de5347f4f1d8ad840e8c4f81af1/users/0f2e48fce4844476bca569acef4a4a62/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:54:19.250111 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d054bc-1378-454f-b82e-7d34c8d32dce None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:19.250111 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d054bc-1378-454f-b82e-7d34c8d32dce None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:19.250111 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d054bc-1378-454f-b82e-7d34c8d32dce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:19.281699 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-e3d054bc-1378-454f-b82e-7d34c8d32dce None None] MFA Rules not processed for user `c13d417208d945c983c2073e84f507f2`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:19.283197 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-e3d054bc-1378-454f-b82e-7d34c8d32dce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:19.315940 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df8bfe55-cbbe-47eb-95e2-d6f23b12d12a admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:19.358594 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1118/2237] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:19 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:54:19.360133 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8634fc-34ec-4c6a-8016-734b456a9e4b None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:19.360710 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8634fc-34ec-4c6a-8016-734b456a9e4b None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:19.360989 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8634fc-34ec-4c6a-8016-734b456a9e4b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:19.380718 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e0fbe68-e0ec-4a21-b645-d0b9f0d818a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:19.381347 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0fbe68-e0ec-4a21-b645-d0b9f0d818a6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:19.381620 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0fbe68-e0ec-4a21-b645-d0b9f0d818a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:19.381914 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0fbe68-e0ec-4a21-b645-d0b9f0d818a6 admin admin] PATH_INFO: `/v3/users/f8aa86aad7d34cc2a707b2cdfe47e8a3` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:19.383140 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e0fbe68-e0ec-4a21-b645-d0b9f0d818a6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f8aa86aad7d34cc2a707b2cdfe47e8a3)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:19.385076 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e0fbe68-e0ec-4a21-b645-d0b9f0d818a6 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:19.389149 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-2c8634fc-34ec-4c6a-8016-734b456a9e4b None None] MFA Rules not processed for user `0f2e48fce4844476bca569acef4a4a62`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:19.390364 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-2c8634fc-34ec-4c6a-8016-734b456a9e4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:19.404989 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:19.414538 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=93,keystone:INSERT=4,keystone:DELETE=3 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:19.441400 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-6e0fbe68-e0ec-4a21-b645-d0b9f0d818a6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f8aa86aad7d34cc2a707b2cdfe47e8a3'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:19.442081 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-6e0fbe68-e0ec-4a21-b645-d0b9f0d818a6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f8aa86aad7d34cc2a707b2cdfe47e8a3'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:19.442705 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-6e0fbe68-e0ec-4a21-b645-d0b9f0d818a6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f8aa86aad7d34cc2a707b2cdfe47e8a3'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:19.457103 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-6e0fbe68-e0ec-4a21-b645-d0b9f0d818a6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f8aa86aad7d34cc2a707b2cdfe47e8a3'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:19.463161 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1119/2238] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:19 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:54:19.464658 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c8634fc-34ec-4c6a-8016-734b456a9e4b None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:19.466210 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-2c8634fc-34ec-4c6a-8016-734b456a9e4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:19.477194 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-6e0fbe68-e0ec-4a21-b645-d0b9f0d818a6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f8aa86aad7d34cc2a707b2cdfe47e8a3'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:19.484945 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c8634fc-34ec-4c6a-8016-734b456a9e4b None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:19.489680 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53be655c-91c3-43e8-bf02-26637be7ca40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:19.490290 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53be655c-91c3-43e8-bf02-26637be7ca40 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:19.490571 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53be655c-91c3-43e8-bf02-26637be7ca40 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:19.490962 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53be655c-91c3-43e8-bf02-26637be7ca40 admin admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:19.491540 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53be655c-91c3-43e8-bf02-26637be7ca40 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:19.492513 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1120/2239] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:54:19 2022] DELETE /identity/v3/users/f8aa86aad7d34cc2a707b2cdfe47e8a3 => generated 0 bytes in 275 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:54:19.493262 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e0fbe68-e0ec-4a21-b645-d0b9f0d818a6 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:19.493442 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53be655c-91c3-43e8-bf02-26637be7ca40 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:19.495415 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-6e0fbe68-e0ec-4a21-b645-d0b9f0d818a6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:19.528208 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1120/2240] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:54:19 2022] POST /identity/v3/projects => generated 389 bytes in 61 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:54:19.528208 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a286bda-8fce-4685-8893-a5c6d3ab6cb8 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:19.528208 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a286bda-8fce-4685-8893-a5c6d3ab6cb8 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:19.528208 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a286bda-8fce-4685-8893-a5c6d3ab6cb8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:19.556982 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-5a286bda-8fce-4685-8893-a5c6d3ab6cb8 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:19.558978 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-5a286bda-8fce-4685-8893-a5c6d3ab6cb8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:19.577239 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e0fbe68-e0ec-4a21-b645-d0b9f0d818a6 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:19.652458 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1121/2241] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:54:19.655126 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06c2d0dc-5e9c-4dde-be44-464a85ebf428 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:19.655819 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c2d0dc-5e9c-4dde-be44-464a85ebf428 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:19.656102 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c2d0dc-5e9c-4dde-be44-464a85ebf428 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:19.656847 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c2d0dc-5e9c-4dde-be44-464a85ebf428 admin admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:19.656992 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a286bda-8fce-4685-8893-a5c6d3ab6cb8 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:19.657092 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06c2d0dc-5e9c-4dde-be44-464a85ebf428 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:19.659428 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06c2d0dc-5e9c-4dde-be44-464a85ebf428 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:19.678074 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a286bda-8fce-4685-8893-a5c6d3ab6cb8 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:19.687144 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25944948-8a18-4aa0-8009-5b60fc748c97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:19.687857 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25944948-8a18-4aa0-8009-5b60fc748c97 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:19.688445 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25944948-8a18-4aa0-8009-5b60fc748c97 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:19.688727 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25944948-8a18-4aa0-8009-5b60fc748c97 admin admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:19.689424 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25944948-8a18-4aa0-8009-5b60fc748c97 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:19.691481 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25944948-8a18-4aa0-8009-5b60fc748c97 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:19.696980 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1121/2242] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:54:19 2022] POST /identity/v3/projects => generated 383 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:54:19.696980 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06c2d0dc-5e9c-4dde-be44-464a85ebf428 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:19.699023 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-06c2d0dc-5e9c-4dde-be44-464a85ebf428 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:19.721000 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1122/2243] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:54:19 2022] POST /identity/v3/users => generated 301 bytes in 68 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:54:19.724510 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25944948-8a18-4aa0-8009-5b60fc748c97 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:19.735083 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25944948-8a18-4aa0-8009-5b60fc748c97 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:19.754992 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:DELETE=7,keystone:SELECT=91,keystone:INSERT=2 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:19.789445 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06c2d0dc-5e9c-4dde-be44-464a85ebf428 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:19.793597 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-860af88b-9bba-4525-8bfb-97a29a9d43af service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:19.794153 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860af88b-9bba-4525-8bfb-97a29a9d43af service neutron] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:19.794532 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860af88b-9bba-4525-8bfb-97a29a9d43af service neutron] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:19.795017 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860af88b-9bba-4525-8bfb-97a29a9d43af service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:19.806417 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ccd5b4-e215-4289-887c-274e3da549ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:19.807872 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ccd5b4-e215-4289-887c-274e3da549ba admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:19.808185 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ccd5b4-e215-4289-887c-274e3da549ba admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:19.808421 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ccd5b4-e215-4289-887c-274e3da549ba admin admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:19.809064 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9ccd5b4-e215-4289-887c-274e3da549ba admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:19.811035 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9ccd5b4-e215-4289-887c-274e3da549ba admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:19.836762 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1123/2244] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:54:19 2022] POST /identity/v3/users => generated 298 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:54:19.837602 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ccd5b4-e215-4289-887c-274e3da549ba admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:19.850135 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ccd5b4-e215-4289-887c-274e3da549ba admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:19.851523 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:19.860782 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f919da9-677a-43e7-bd46-b72fe92f12e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:19.862591 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f919da9-677a-43e7-bd46-b72fe92f12e0 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:19.862591 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f919da9-677a-43e7-bd46-b72fe92f12e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:19.862591 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f919da9-677a-43e7-bd46-b72fe92f12e0 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:19.863261 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f919da9-677a-43e7-bd46-b72fe92f12e0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:19.864965 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f919da9-677a-43e7-bd46-b72fe92f12e0 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:19.875012 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-860af88b-9bba-4525-8bfb-97a29a9d43af service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:19.876321 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1124/2245] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:54:19 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:54:19.879585 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f919da9-677a-43e7-bd46-b72fe92f12e0 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:19.879679 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-860af88b-9bba-4525-8bfb-97a29a9d43af service neutron] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:19.889617 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f919da9-677a-43e7-bd46-b72fe92f12e0 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:19.892323 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1122/2246] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:54:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 197 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:54:19.893358 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-860af88b-9bba-4525-8bfb-97a29a9d43af service neutron] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:19.906004 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-987f5aa6-d29d-4bd1-869d-93cfebddffc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:19.907061 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987f5aa6-d29d-4bd1-869d-93cfebddffc3 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:19.907650 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987f5aa6-d29d-4bd1-869d-93cfebddffc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:19.908098 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987f5aa6-d29d-4bd1-869d-93cfebddffc3 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:19.908839 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-860af88b-9bba-4525-8bfb-97a29a9d43af service neutron] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:19.909057 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-987f5aa6-d29d-4bd1-869d-93cfebddffc3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:19.911881 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-987f5aa6-d29d-4bd1-869d-93cfebddffc3 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:19.919322 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db5c49d-529e-4cb2-92ce-7eb4a60db8ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:19.920246 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db5c49d-529e-4cb2-92ce-7eb4a60db8ce admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:19.920605 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db5c49d-529e-4cb2-92ce-7eb4a60db8ce admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:19.921018 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db5c49d-529e-4cb2-92ce-7eb4a60db8ce admin admin] PATH_INFO: `/v3/projects/9495291cc9c244099964027277365883/users/8a27cbc53aa84afaaa2b2b97aba441d1/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:19.932462 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1125/2247] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:54:19 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:54:19.937180 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-987f5aa6-d29d-4bd1-869d-93cfebddffc3 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:19.940216 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2db5c49d-529e-4cb2-92ce-7eb4a60db8ce admin admin] RBAC: Authorizing `identity:create_grant(project_id=9495291cc9c244099964027277365883, user_id=8a27cbc53aa84afaaa2b2b97aba441d1, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:19.943065 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2db5c49d-529e-4cb2-92ce-7eb4a60db8ce admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:19.956668 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-987f5aa6-d29d-4bd1-869d-93cfebddffc3 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:19.969995 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52d5023a-b5bd-4e42-a209-3b9c4999002f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:19.969995 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d5023a-b5bd-4e42-a209-3b9c4999002f admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:19.969995 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d5023a-b5bd-4e42-a209-3b9c4999002f admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:19.969995 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d5023a-b5bd-4e42-a209-3b9c4999002f admin admin] PATH_INFO: `/v3/projects/61aa2fdb893140b3a0fc3ffbe223970f/users/501c024d392143d6b3ea7448916f470f/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:19.980945 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1123/2248] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:54:19 2022] PUT /identity/v3/projects/9495291cc9c244099964027277365883/users/8a27cbc53aa84afaaa2b2b97aba441d1/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 88 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:54:19.987800 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db5c49d-529e-4cb2-92ce-7eb4a60db8ce admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:19.990592 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52d5023a-b5bd-4e42-a209-3b9c4999002f admin admin] RBAC: Authorizing `identity:create_grant(project_id=61aa2fdb893140b3a0fc3ffbe223970f, user_id=501c024d392143d6b3ea7448916f470f, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:19.994084 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52d5023a-b5bd-4e42-a209-3b9c4999002f admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:20.014010 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1126/2249] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:54:19 2022] PUT /identity/v3/projects/61aa2fdb893140b3a0fc3ffbe223970f/users/501c024d392143d6b3ea7448916f470f/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 78 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:54:20.018214 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52d5023a-b5bd-4e42-a209-3b9c4999002f admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:20.148714 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db5c49d-529e-4cb2-92ce-7eb4a60db8ce admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:20.151559 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52d5023a-b5bd-4e42-a209-3b9c4999002f admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:20.153748 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be36789d-49c8-4297-9880-6feb8f5845b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:20.154317 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be36789d-49c8-4297-9880-6feb8f5845b9 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:20.154874 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be36789d-49c8-4297-9880-6feb8f5845b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:20.155039 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be36789d-49c8-4297-9880-6feb8f5845b9 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:20.156207 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be36789d-49c8-4297-9880-6feb8f5845b9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:20.157394 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11191655-a1ad-4d2c-a6b6-4b01ba5821f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:20.157952 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11191655-a1ad-4d2c-a6b6-4b01ba5821f2 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:20.158140 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be36789d-49c8-4297-9880-6feb8f5845b9 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:20.158431 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11191655-a1ad-4d2c-a6b6-4b01ba5821f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:20.159026 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11191655-a1ad-4d2c-a6b6-4b01ba5821f2 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:20.160371 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11191655-a1ad-4d2c-a6b6-4b01ba5821f2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:20.168911 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11191655-a1ad-4d2c-a6b6-4b01ba5821f2 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:20.171199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1124/2250] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:54:19 2022] GET /identity/v3/roles => generated 1420 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 11 00:54:20.175044 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be36789d-49c8-4297-9880-6feb8f5845b9 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:20.180384 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1127/2251] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:54:20 2022] GET /identity/v3/roles => generated 1420 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 11 00:54:20.184514 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11191655-a1ad-4d2c-a6b6-4b01ba5821f2 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:20.186660 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be36789d-49c8-4297-9880-6feb8f5845b9 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:20.192721 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e68b5ded-c730-443a-bfbe-962fec1151b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:20.193602 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68b5ded-c730-443a-bfbe-962fec1151b3 admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:20.194322 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68b5ded-c730-443a-bfbe-962fec1151b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:20.194810 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68b5ded-c730-443a-bfbe-962fec1151b3 admin admin] PATH_INFO: `/v3/projects/9495291cc9c244099964027277365883/users/8a27cbc53aa84afaaa2b2b97aba441d1/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:20.195640 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11191655-a1ad-4d2c-a6b6-4b01ba5821f2 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:20.197200 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e68b5ded-c730-443a-bfbe-962fec1151b3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9495291cc9c244099964027277365883, user_id=8a27cbc53aa84afaaa2b2b97aba441d1, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:20.200897 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e68b5ded-c730-443a-bfbe-962fec1151b3 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:20.203013 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b712b7c-5af9-4b0f-8f1b-ecaf13153996 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:20.204029 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b712b7c-5af9-4b0f-8f1b-ecaf13153996 admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:20.204287 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b712b7c-5af9-4b0f-8f1b-ecaf13153996 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:20.204582 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b712b7c-5af9-4b0f-8f1b-ecaf13153996 admin admin] PATH_INFO: `/v3/projects/61aa2fdb893140b3a0fc3ffbe223970f/users/501c024d392143d6b3ea7448916f470f/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:20.207094 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b712b7c-5af9-4b0f-8f1b-ecaf13153996 admin admin] RBAC: Authorizing `identity:create_grant(project_id=61aa2fdb893140b3a0fc3ffbe223970f, user_id=501c024d392143d6b3ea7448916f470f, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:20.209691 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b712b7c-5af9-4b0f-8f1b-ecaf13153996 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:20.214585 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1125/2252] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:54:20 2022] PUT /identity/v3/projects/9495291cc9c244099964027277365883/users/8a27cbc53aa84afaaa2b2b97aba441d1/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:54:20.222348 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1128/2253] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:54:20 2022] PUT /identity/v3/projects/61aa2fdb893140b3a0fc3ffbe223970f/users/501c024d392143d6b3ea7448916f470f/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:54:20.224829 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971683cd-f128-4dc7-9c5d-9a12ffa9cd97 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:20.225119 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971683cd-f128-4dc7-9c5d-9a12ffa9cd97 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:20.225395 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971683cd-f128-4dc7-9c5d-9a12ffa9cd97 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:20.233812 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e8a791-a485-451a-9cff-609894e810bf None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:20.234100 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e8a791-a485-451a-9cff-609894e810bf None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:20.234379 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e8a791-a485-451a-9cff-609894e810bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:20.264941 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-971683cd-f128-4dc7-9c5d-9a12ffa9cd97 None None] MFA Rules not processed for user `8a27cbc53aa84afaaa2b2b97aba441d1`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:20.266654 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-971683cd-f128-4dc7-9c5d-9a12ffa9cd97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:20.270107 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-87e8a791-a485-451a-9cff-609894e810bf None None] MFA Rules not processed for user `501c024d392143d6b3ea7448916f470f`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:20.277742 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-87e8a791-a485-451a-9cff-609894e810bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:20.312374 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:20.355854 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1129/2254] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:20 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:54:20.356875 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e8a791-a485-451a-9cff-609894e810bf None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:20.358614 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-87e8a791-a485-451a-9cff-609894e810bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:20.365057 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1126/2255] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:20 2022] POST /identity/v3/auth/tokens => generated 3333 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:54:20.367733 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971683cd-f128-4dc7-9c5d-9a12ffa9cd97 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:20.369291 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-971683cd-f128-4dc7-9c5d-9a12ffa9cd97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:20.394049 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:20.428653 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971683cd-f128-4dc7-9c5d-9a12ffa9cd97 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:20.429467 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e8a791-a485-451a-9cff-609894e810bf None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:20.432945 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5872e24-6f42-4d3c-a72b-48c69df10f1a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:20.433568 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5872e24-6f42-4d3c-a72b-48c69df10f1a service cinder] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:20.433843 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5872e24-6f42-4d3c-a72b-48c69df10f1a service cinder] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:20.434112 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5872e24-6f42-4d3c-a72b-48c69df10f1a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:20.434913 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae2bfef2-098d-41e0-9ab1-74dd051d32d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:20.435537 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2bfef2-098d-41e0-9ab1-74dd051d32d9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:20.435812 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2bfef2-098d-41e0-9ab1-74dd051d32d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:20.435963 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-d5872e24-6f42-4d3c-a72b-48c69df10f1a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:20.436065 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2bfef2-098d-41e0-9ab1-74dd051d32d9 admin admin] PATH_INFO: `/v3/projects/7656f24070314d4480d607f85937a84d` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:20.437177 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae2bfef2-098d-41e0-9ab1-74dd051d32d9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7656f24070314d4480d607f85937a84d)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:20.439093 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae2bfef2-098d-41e0-9ab1-74dd051d32d9 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:20.488672 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-ae2bfef2-098d-41e0-9ab1-74dd051d32d9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7656f24070314d4480d607f85937a84d', 'request_id': 'req-ae2bfef2-098d-41e0-9ab1-74dd051d32d9'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:20.495902 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-ae2bfef2-098d-41e0-9ab1-74dd051d32d9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7656f24070314d4480d607f85937a84d', 'request_id': 'req-ae2bfef2-098d-41e0-9ab1-74dd051d32d9'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:20.502680 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5872e24-6f42-4d3c-a72b-48c69df10f1a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:20.504074 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-ae2bfef2-098d-41e0-9ab1-74dd051d32d9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7656f24070314d4480d607f85937a84d', 'request_id': 'req-ae2bfef2-098d-41e0-9ab1-74dd051d32d9'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:20.504228 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5872e24-6f42-4d3c-a72b-48c69df10f1a service cinder] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:20.514028 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1127/2256] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:54:20 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 11 00:54:20.514957 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5872e24-6f42-4d3c-a72b-48c69df10f1a service cinder] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:20.516313 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-d5872e24-6f42-4d3c-a72b-48c69df10f1a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:20.517288 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-ae2bfef2-098d-41e0-9ab1-74dd051d32d9 admin admin] The token cache is being invalidate because project 7656f24070314d4480d607f85937a84d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87924) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:54:20.517583 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-ae2bfef2-098d-41e0-9ab1-74dd051d32d9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:20.518248 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-ae2bfef2-098d-41e0-9ab1-74dd051d32d9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:20.519995 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1130/2257] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:54:20 2022] DELETE /identity/v3/projects/7656f24070314d4480d607f85937a84d => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:54:20.523648 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae2bfef2-098d-41e0-9ab1-74dd051d32d9 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:20.525584 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-ae2bfef2-098d-41e0-9ab1-74dd051d32d9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:20.527708 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5872e24-6f42-4d3c-a72b-48c69df10f1a service cinder] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:20.596988 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5029b90-9ba0-4930-b5ca-f579097a81da service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:20.597614 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5029b90-9ba0-4930-b5ca-f579097a81da service cinder] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:20.598057 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5029b90-9ba0-4930-b5ca-f579097a81da service cinder] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:20.598318 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5029b90-9ba0-4930-b5ca-f579097a81da service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:20.600289 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-b5029b90-9ba0-4930-b5ca-f579097a81da service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:20.601385 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae2bfef2-098d-41e0-9ab1-74dd051d32d9 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:20.607254 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-524ffa23-157c-4612-9d57-d54768ae8538 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:20.607919 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-524ffa23-157c-4612-9d57-d54768ae8538 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:20.608296 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-524ffa23-157c-4612-9d57-d54768ae8538 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:20.608468 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-524ffa23-157c-4612-9d57-d54768ae8538 admin admin] PATH_INFO: `/v3/users/a12f234cac6141dd99e62168c3368b3a` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:20.611450 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-524ffa23-157c-4612-9d57-d54768ae8538 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a12f234cac6141dd99e62168c3368b3a)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:20.613369 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-524ffa23-157c-4612-9d57-d54768ae8538 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:20.639302 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:20.669044 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-524ffa23-157c-4612-9d57-d54768ae8538 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a12f234cac6141dd99e62168c3368b3a'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:20.669730 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-524ffa23-157c-4612-9d57-d54768ae8538 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a12f234cac6141dd99e62168c3368b3a'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:20.670368 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-524ffa23-157c-4612-9d57-d54768ae8538 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a12f234cac6141dd99e62168c3368b3a'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:20.676389 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5029b90-9ba0-4930-b5ca-f579097a81da service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:20.678175 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5029b90-9ba0-4930-b5ca-f579097a81da service cinder] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:20.679156 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-524ffa23-157c-4612-9d57-d54768ae8538 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a12f234cac6141dd99e62168c3368b3a'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:20.681000 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:20.690137 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1128/2258] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:54:20 2022] GET /identity/v3/auth/tokens => generated 3333 bytes in 175 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:54:20.696975 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-524ffa23-157c-4612-9d57-d54768ae8538 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a12f234cac6141dd99e62168c3368b3a'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:20.711601 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1131/2259] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:54:20 2022] DELETE /identity/v3/users/a12f234cac6141dd99e62168c3368b3a => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:54:20.913024 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5029b90-9ba0-4930-b5ca-f579097a81da service cinder] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:20.914612 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-b5029b90-9ba0-4930-b5ca-f579097a81da service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:20.957218 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5029b90-9ba0-4930-b5ca-f579097a81da service cinder] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:21.084333 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746c3cd4-f1c1-45b3-bdfc-365283712963 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:21.084984 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746c3cd4-f1c1-45b3-bdfc-365283712963 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:21.085265 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746c3cd4-f1c1-45b3-bdfc-365283712963 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:21.085537 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746c3cd4-f1c1-45b3-bdfc-365283712963 admin admin] PATH_INFO: `/v3/users/c13d417208d945c983c2073e84f507f2` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:21.086740 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-746c3cd4-f1c1-45b3-bdfc-365283712963 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c13d417208d945c983c2073e84f507f2)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:21.089385 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-746c3cd4-f1c1-45b3-bdfc-365283712963 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:21.141061 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-746c3cd4-f1c1-45b3-bdfc-365283712963 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c13d417208d945c983c2073e84f507f2'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:21.141733 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-746c3cd4-f1c1-45b3-bdfc-365283712963 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c13d417208d945c983c2073e84f507f2'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:21.142454 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-746c3cd4-f1c1-45b3-bdfc-365283712963 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c13d417208d945c983c2073e84f507f2'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:21.152482 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-746c3cd4-f1c1-45b3-bdfc-365283712963 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c13d417208d945c983c2073e84f507f2'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:21.189536 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-524ffa23-157c-4612-9d57-d54768ae8538 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:21.191294 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-524ffa23-157c-4612-9d57-d54768ae8538 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:21.195406 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-746c3cd4-f1c1-45b3-bdfc-365283712963 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c13d417208d945c983c2073e84f507f2'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:21.207947 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-524ffa23-157c-4612-9d57-d54768ae8538 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:21.214166 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ccc6e09-32a8-40fa-bafe-9ccaa92ced40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:21.215226 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ccc6e09-32a8-40fa-bafe-9ccaa92ced40 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:21.215522 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ccc6e09-32a8-40fa-bafe-9ccaa92ced40 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:21.215780 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ccc6e09-32a8-40fa-bafe-9ccaa92ced40 admin admin] PATH_INFO: `/v3/projects/c2d6e7dabf6445539f5f783ec7991cee` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:21.216948 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ccc6e09-32a8-40fa-bafe-9ccaa92ced40 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c2d6e7dabf6445539f5f783ec7991cee)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:21.218826 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1129/2260] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:54:20 2022] DELETE /identity/v3/users/c13d417208d945c983c2073e84f507f2 => generated 0 bytes in 307 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:54:21.218950 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ccc6e09-32a8-40fa-bafe-9ccaa92ced40 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:21.223555 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff07ac39-f2e9-4e9b-b1a0-84c84ccfd1fe None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:21.224048 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff07ac39-f2e9-4e9b-b1a0-84c84ccfd1fe None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:21.224466 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff07ac39-f2e9-4e9b-b1a0-84c84ccfd1fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:21.247188 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-ff07ac39-f2e9-4e9b-b1a0-84c84ccfd1fe None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:21.248514 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-ff07ac39-f2e9-4e9b-b1a0-84c84ccfd1fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:21.280133 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-9ccc6e09-32a8-40fa-bafe-9ccaa92ced40 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c2d6e7dabf6445539f5f783ec7991cee', 'request_id': 'req-9ccc6e09-32a8-40fa-bafe-9ccaa92ced40'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:21.287451 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-9ccc6e09-32a8-40fa-bafe-9ccaa92ced40 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c2d6e7dabf6445539f5f783ec7991cee', 'request_id': 'req-9ccc6e09-32a8-40fa-bafe-9ccaa92ced40'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:21.296458 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-9ccc6e09-32a8-40fa-bafe-9ccaa92ced40 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c2d6e7dabf6445539f5f783ec7991cee', 'request_id': 'req-9ccc6e09-32a8-40fa-bafe-9ccaa92ced40'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:21.312369 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:21.317480 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-9ccc6e09-32a8-40fa-bafe-9ccaa92ced40 admin admin] The token cache is being invalidate because project c2d6e7dabf6445539f5f783ec7991cee was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87924) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:54:21.317999 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-9ccc6e09-32a8-40fa-bafe-9ccaa92ced40 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:21.318747 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-9ccc6e09-32a8-40fa-bafe-9ccaa92ced40 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:21.323235 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1132/2261] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:54:21 2022] DELETE /identity/v3/projects/c2d6e7dabf6445539f5f783ec7991cee => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:54:21.325046 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ccc6e09-32a8-40fa-bafe-9ccaa92ced40 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:21.326860 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1130/2262] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:54:21.326981 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-9ccc6e09-32a8-40fa-bafe-9ccaa92ced40 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:21.339424 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff07ac39-f2e9-4e9b-b1a0-84c84ccfd1fe None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:21.341646 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-ff07ac39-f2e9-4e9b-b1a0-84c84ccfd1fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:21.415725 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff07ac39-f2e9-4e9b-b1a0-84c84ccfd1fe None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:21.435449 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ccc6e09-32a8-40fa-bafe-9ccaa92ced40 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:21.440773 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfccc2e3-b1d6-4be0-8b23-b13dd9ad87b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:21.441469 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfccc2e3-b1d6-4be0-8b23-b13dd9ad87b8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:21.441810 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfccc2e3-b1d6-4be0-8b23-b13dd9ad87b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:21.442133 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfccc2e3-b1d6-4be0-8b23-b13dd9ad87b8 admin admin] PATH_INFO: `/v3/users/25ef3d2ef08f4d85b60d923ec35ae092` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:21.443330 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfccc2e3-b1d6-4be0-8b23-b13dd9ad87b8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=25ef3d2ef08f4d85b60d923ec35ae092)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:21.445268 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfccc2e3-b1d6-4be0-8b23-b13dd9ad87b8 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:21.477868 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d167c58b-ea02-4dd2-ae5b-3ccaed0347aa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:21.478453 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d167c58b-ea02-4dd2-ae5b-3ccaed0347aa service neutron] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:21.478759 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d167c58b-ea02-4dd2-ae5b-3ccaed0347aa service neutron] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:21.479553 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d167c58b-ea02-4dd2-ae5b-3ccaed0347aa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:21.490196 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d167c58b-ea02-4dd2-ae5b-3ccaed0347aa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:21.491720 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d167c58b-ea02-4dd2-ae5b-3ccaed0347aa service neutron] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:21.502528 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1131/2263] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:54:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:54:21.514336 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-dfccc2e3-b1d6-4be0-8b23-b13dd9ad87b8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '25ef3d2ef08f4d85b60d923ec35ae092'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:21.515129 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-dfccc2e3-b1d6-4be0-8b23-b13dd9ad87b8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '25ef3d2ef08f4d85b60d923ec35ae092'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:21.515761 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-dfccc2e3-b1d6-4be0-8b23-b13dd9ad87b8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '25ef3d2ef08f4d85b60d923ec35ae092'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:21.523393 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-dfccc2e3-b1d6-4be0-8b23-b13dd9ad87b8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '25ef3d2ef08f4d85b60d923ec35ae092'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:21.528502 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:21.540016 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-dfccc2e3-b1d6-4be0-8b23-b13dd9ad87b8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '25ef3d2ef08f4d85b60d923ec35ae092'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:21.557256 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1133/2264] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:54:21 2022] DELETE /identity/v3/users/25ef3d2ef08f4d85b60d923ec35ae092 => generated 0 bytes in 233 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:54:21.851632 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d167c58b-ea02-4dd2-ae5b-3ccaed0347aa service neutron] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:21.853151 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-d167c58b-ea02-4dd2-ae5b-3ccaed0347aa service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:21.868735 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d167c58b-ea02-4dd2-ae5b-3ccaed0347aa service neutron] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:21.897406 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfccc2e3-b1d6-4be0-8b23-b13dd9ad87b8 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:21.899093 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-dfccc2e3-b1d6-4be0-8b23-b13dd9ad87b8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:21.942308 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9330e165-6111-431c-b320-fbb0131ae0cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:21.942802 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9330e165-6111-431c-b320-fbb0131ae0cc admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:21.943099 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9330e165-6111-431c-b320-fbb0131ae0cc admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:21.943370 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9330e165-6111-431c-b320-fbb0131ae0cc admin admin] PATH_INFO: `/v3/projects/b78cf66aead646ff894292a407f7ca73` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:21.944513 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9330e165-6111-431c-b320-fbb0131ae0cc admin admin] RBAC: Authorizing `identity:delete_project(project_id=b78cf66aead646ff894292a407f7ca73)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:21.946471 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9330e165-6111-431c-b320-fbb0131ae0cc admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:21.974724 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfccc2e3-b1d6-4be0-8b23-b13dd9ad87b8 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:21.980066 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28aaef7d-7b30-4ea2-ae60-4c06156338d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:21.980970 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28aaef7d-7b30-4ea2-ae60-4c06156338d8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:21.981357 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28aaef7d-7b30-4ea2-ae60-4c06156338d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:21.981664 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28aaef7d-7b30-4ea2-ae60-4c06156338d8 admin admin] PATH_INFO: `/v3/projects/7c878c184e224223b0a40568a615614a` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:21.983091 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28aaef7d-7b30-4ea2-ae60-4c06156338d8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7c878c184e224223b0a40568a615614a)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:21.985378 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28aaef7d-7b30-4ea2-ae60-4c06156338d8 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:22.009396 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-9330e165-6111-431c-b320-fbb0131ae0cc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b78cf66aead646ff894292a407f7ca73', 'request_id': 'req-9330e165-6111-431c-b320-fbb0131ae0cc'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:22.010938 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:22.016740 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-9330e165-6111-431c-b320-fbb0131ae0cc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b78cf66aead646ff894292a407f7ca73', 'request_id': 'req-9330e165-6111-431c-b320-fbb0131ae0cc'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:22.031306 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-9330e165-6111-431c-b320-fbb0131ae0cc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b78cf66aead646ff894292a407f7ca73', 'request_id': 'req-9330e165-6111-431c-b320-fbb0131ae0cc'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:22.045136 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-28aaef7d-7b30-4ea2-ae60-4c06156338d8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7c878c184e224223b0a40568a615614a', 'request_id': 'req-28aaef7d-7b30-4ea2-ae60-4c06156338d8'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:22.049248 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-9330e165-6111-431c-b320-fbb0131ae0cc admin admin] The token cache is being invalidate because project b78cf66aead646ff894292a407f7ca73 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87923) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:54:22.049539 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-9330e165-6111-431c-b320-fbb0131ae0cc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:22.050226 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-9330e165-6111-431c-b320-fbb0131ae0cc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:22.051985 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1132/2265] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:54:21 2022] DELETE /identity/v3/projects/b78cf66aead646ff894292a407f7ca73 => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:54:22.053709 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-28aaef7d-7b30-4ea2-ae60-4c06156338d8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7c878c184e224223b0a40568a615614a', 'request_id': 'req-28aaef7d-7b30-4ea2-ae60-4c06156338d8'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:22.055864 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9330e165-6111-431c-b320-fbb0131ae0cc admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:22.057364 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-9330e165-6111-431c-b320-fbb0131ae0cc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:22.062885 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-28aaef7d-7b30-4ea2-ae60-4c06156338d8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7c878c184e224223b0a40568a615614a', 'request_id': 'req-28aaef7d-7b30-4ea2-ae60-4c06156338d8'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:22.077489 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-28aaef7d-7b30-4ea2-ae60-4c06156338d8 admin admin] The token cache is being invalidate because project 7c878c184e224223b0a40568a615614a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87924) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:54:22.077785 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-28aaef7d-7b30-4ea2-ae60-4c06156338d8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:22.078715 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-28aaef7d-7b30-4ea2-ae60-4c06156338d8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:22.083270 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1134/2266] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:54:21 2022] DELETE /identity/v3/projects/7c878c184e224223b0a40568a615614a => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:54:22.085929 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d3b86a-a0b1-44fd-add3-50558d724db4 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:22.085929 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d3b86a-a0b1-44fd-add3-50558d724db4 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:22.086222 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d3b86a-a0b1-44fd-add3-50558d724db4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:22.118105 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-b6d3b86a-a0b1-44fd-add3-50558d724db4 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:22.119357 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-b6d3b86a-a0b1-44fd-add3-50558d724db4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:22.140526 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9330e165-6111-431c-b320-fbb0131ae0cc admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:22.202911 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1135/2267] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:54:22.202911 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981eb512-2964-4840-bcfd-92b283e1b6c0 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:22.202911 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981eb512-2964-4840-bcfd-92b283e1b6c0 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:22.202911 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981eb512-2964-4840-bcfd-92b283e1b6c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:22.215202 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb52ace2-098f-4229-a4e7-6e2e12d2d2e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:22.216328 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb52ace2-098f-4229-a4e7-6e2e12d2d2e6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:22.216633 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb52ace2-098f-4229-a4e7-6e2e12d2d2e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:22.217056 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb52ace2-098f-4229-a4e7-6e2e12d2d2e6 admin admin] PATH_INFO: `/v3/users/8a27cbc53aa84afaaa2b2b97aba441d1` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:22.218423 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb52ace2-098f-4229-a4e7-6e2e12d2d2e6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8a27cbc53aa84afaaa2b2b97aba441d1)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:22.220711 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb52ace2-098f-4229-a4e7-6e2e12d2d2e6 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:22.237653 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-981eb512-2964-4840-bcfd-92b283e1b6c0 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:22.239016 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-981eb512-2964-4840-bcfd-92b283e1b6c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:22.252203 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1136/2268] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 52 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:54:22.252203 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-981eb512-2964-4840-bcfd-92b283e1b6c0 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:22.254352 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-981eb512-2964-4840-bcfd-92b283e1b6c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:22.260251 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:DELETE=10,keystone:SELECT=88,keystone:INSERT=2 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:22.274745 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-eb52ace2-098f-4229-a4e7-6e2e12d2d2e6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8a27cbc53aa84afaaa2b2b97aba441d1'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:22.275525 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-eb52ace2-098f-4229-a4e7-6e2e12d2d2e6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8a27cbc53aa84afaaa2b2b97aba441d1'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:22.276554 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-eb52ace2-098f-4229-a4e7-6e2e12d2d2e6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8a27cbc53aa84afaaa2b2b97aba441d1'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:22.289764 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-eb52ace2-098f-4229-a4e7-6e2e12d2d2e6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8a27cbc53aa84afaaa2b2b97aba441d1'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:22.305374 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-eb52ace2-098f-4229-a4e7-6e2e12d2d2e6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8a27cbc53aa84afaaa2b2b97aba441d1'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:22.320329 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1133/2269] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:54:22 2022] DELETE /identity/v3/users/8a27cbc53aa84afaaa2b2b97aba441d1 => generated 0 bytes in 265 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:54:22.326426 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb52ace2-098f-4229-a4e7-6e2e12d2d2e6 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:22.328024 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-981eb512-2964-4840-bcfd-92b283e1b6c0 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:22.328352 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-eb52ace2-098f-4229-a4e7-6e2e12d2d2e6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:22.335625 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:22.390669 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16ec9a22-e85a-4848-9044-2df5960f96f9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:22.390943 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ec9a22-e85a-4848-9044-2df5960f96f9 service cinder] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:22.391076 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ec9a22-e85a-4848-9044-2df5960f96f9 service cinder] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:22.391485 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ec9a22-e85a-4848-9044-2df5960f96f9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:22.393968 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-16ec9a22-e85a-4848-9044-2df5960f96f9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:22.424337 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb52ace2-098f-4229-a4e7-6e2e12d2d2e6 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:22.430059 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48bd1d00-d9fc-4334-98cf-fa57907966c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:22.440365 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48bd1d00-d9fc-4334-98cf-fa57907966c0 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:22.440655 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48bd1d00-d9fc-4334-98cf-fa57907966c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:22.440935 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48bd1d00-d9fc-4334-98cf-fa57907966c0 admin admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:22.441616 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48bd1d00-d9fc-4334-98cf-fa57907966c0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:54:22.441928 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48bd1d00-d9fc-4334-98cf-fa57907966c0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:22.449620 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48bd1d00-d9fc-4334-98cf-fa57907966c0 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:22.481308 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1134/2270] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 00:54:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:54:22.481904 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16ec9a22-e85a-4848-9044-2df5960f96f9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:22.482275 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16ec9a22-e85a-4848-9044-2df5960f96f9 service cinder] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:22.483100 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48bd1d00-d9fc-4334-98cf-fa57907966c0 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:22.495206 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1137/2271] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:54:22 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 245 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:54:22.500066 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48bd1d00-d9fc-4334-98cf-fa57907966c0 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:22.505665 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6265351-8138-435c-8681-b752e5a41837 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:22.506461 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6265351-8138-435c-8681-b752e5a41837 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:22.506961 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6265351-8138-435c-8681-b752e5a41837 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:22.507418 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6265351-8138-435c-8681-b752e5a41837 admin admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:22.508175 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6265351-8138-435c-8681-b752e5a41837 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:22.510334 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6265351-8138-435c-8681-b752e5a41837 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:22.548885 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1135/2272] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:54:22 2022] POST /identity/v3/projects => generated 375 bytes in 66 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:54:22.553544 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16ec9a22-e85a-4848-9044-2df5960f96f9 service cinder] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:22.576093 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16ec9a22-e85a-4848-9044-2df5960f96f9 service cinder] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:22.696423 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4546ce0-3cb5-4baa-b35d-3cddd19e1258 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:22.701824 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4546ce0-3cb5-4baa-b35d-3cddd19e1258 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:22.702336 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4546ce0-3cb5-4baa-b35d-3cddd19e1258 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:22.702836 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4546ce0-3cb5-4baa-b35d-3cddd19e1258 admin admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:22.703698 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4546ce0-3cb5-4baa-b35d-3cddd19e1258 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:22.706090 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4546ce0-3cb5-4baa-b35d-3cddd19e1258 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:22.756934 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1138/2273] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:54:22 2022] POST /identity/v3/users => generated 288 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:54:22.761356 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6265351-8138-435c-8681-b752e5a41837 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:22.782320 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4546ce0-3cb5-4baa-b35d-3cddd19e1258 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:22.783057 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6265351-8138-435c-8681-b752e5a41837 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:22.786765 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-b4546ce0-3cb5-4baa-b35d-3cddd19e1258 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:22.795646 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b0736a-e0e6-4b4d-9ca7-fb316e825437 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:22.796697 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b0736a-e0e6-4b4d-9ca7-fb316e825437 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:22.797119 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b0736a-e0e6-4b4d-9ca7-fb316e825437 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:22.797230 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b0736a-e0e6-4b4d-9ca7-fb316e825437 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:22.798081 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05b0736a-e0e6-4b4d-9ca7-fb316e825437 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:22.803512 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05b0736a-e0e6-4b4d-9ca7-fb316e825437 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:22.805443 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:22.824682 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4546ce0-3cb5-4baa-b35d-3cddd19e1258 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:22.825900 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1136/2274] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:54:22 2022] GET /identity/v3/roles => generated 1420 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:54:22.827512 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b0736a-e0e6-4b4d-9ca7-fb316e825437 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:22.834798 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b271a0d-ece5-4b57-9255-f63a60ec93b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:22.835628 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b271a0d-ece5-4b57-9255-f63a60ec93b6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:22.835916 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b271a0d-ece5-4b57-9255-f63a60ec93b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:22.836266 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b271a0d-ece5-4b57-9255-f63a60ec93b6 admin admin] PATH_INFO: `/v3/projects/9495291cc9c244099964027277365883` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:22.837718 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b271a0d-ece5-4b57-9255-f63a60ec93b6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9495291cc9c244099964027277365883)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:22.840426 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b271a0d-ece5-4b57-9255-f63a60ec93b6 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:22.846626 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b0736a-e0e6-4b4d-9ca7-fb316e825437 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:22.851870 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f84756-00d8-40b6-8be9-8b60dfab7a0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:22.852707 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f84756-00d8-40b6-8be9-8b60dfab7a0a admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:22.852987 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f84756-00d8-40b6-8be9-8b60dfab7a0a admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:22.853260 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f84756-00d8-40b6-8be9-8b60dfab7a0a admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:22.854549 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f84756-00d8-40b6-8be9-8b60dfab7a0a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:22.856364 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f84756-00d8-40b6-8be9-8b60dfab7a0a admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:22.867169 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1137/2275] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:54:22 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:54:22.871632 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f84756-00d8-40b6-8be9-8b60dfab7a0a admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:22.883027 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f84756-00d8-40b6-8be9-8b60dfab7a0a admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:22.900249 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-0b271a0d-ece5-4b57-9255-f63a60ec93b6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9495291cc9c244099964027277365883', 'request_id': 'req-0b271a0d-ece5-4b57-9255-f63a60ec93b6'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:22.911104 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-0b271a0d-ece5-4b57-9255-f63a60ec93b6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9495291cc9c244099964027277365883', 'request_id': 'req-0b271a0d-ece5-4b57-9255-f63a60ec93b6'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:22.919890 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-0b271a0d-ece5-4b57-9255-f63a60ec93b6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9495291cc9c244099964027277365883', 'request_id': 'req-0b271a0d-ece5-4b57-9255-f63a60ec93b6'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:22.932601 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-0b271a0d-ece5-4b57-9255-f63a60ec93b6 admin admin] The token cache is being invalidate because project 9495291cc9c244099964027277365883 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87924) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:54:22.932981 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-0b271a0d-ece5-4b57-9255-f63a60ec93b6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:22.933661 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-0b271a0d-ece5-4b57-9255-f63a60ec93b6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:22.935217 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1139/2276] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:54:22 2022] DELETE /identity/v3/projects/9495291cc9c244099964027277365883 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:54:22.939809 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66446bb8-ea1e-443a-8d0e-88f931db8c17 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:22.940095 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66446bb8-ea1e-443a-8d0e-88f931db8c17 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:22.940372 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66446bb8-ea1e-443a-8d0e-88f931db8c17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:22.946809 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e0def48-701b-44b9-845c-e27fbe279bdb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:22.947488 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e0def48-701b-44b9-845c-e27fbe279bdb admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:22.948471 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e0def48-701b-44b9-845c-e27fbe279bdb admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:22.948741 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e0def48-701b-44b9-845c-e27fbe279bdb admin admin] PATH_INFO: `/v3/projects/cdf84ece5642437a9b28d272e9ff5df2/users/17d4eb55809b49039a6cca481552cef2/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:22.961843 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-66446bb8-ea1e-443a-8d0e-88f931db8c17 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:22.966277 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-66446bb8-ea1e-443a-8d0e-88f931db8c17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:22.974634 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e0def48-701b-44b9-845c-e27fbe279bdb admin admin] RBAC: Authorizing `identity:create_grant(project_id=cdf84ece5642437a9b28d272e9ff5df2, user_id=17d4eb55809b49039a6cca481552cef2, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:22.977574 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e0def48-701b-44b9-845c-e27fbe279bdb admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:22.981400 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1140/2277] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:54:22.982655 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7448bd4e-a4ea-4de8-acc4-4bda3f868e3e None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:22.982986 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7448bd4e-a4ea-4de8-acc4-4bda3f868e3e None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:22.983335 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7448bd4e-a4ea-4de8-acc4-4bda3f868e3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:22.996884 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1138/2278] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:54:22 2022] PUT /identity/v3/projects/cdf84ece5642437a9b28d272e9ff5df2/users/17d4eb55809b49039a6cca481552cef2/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:54:23.008225 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f051f926-c019-4472-b5b9-a6f7873ee0c3 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:23.008518 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f051f926-c019-4472-b5b9-a6f7873ee0c3 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:23.008794 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f051f926-c019-4472-b5b9-a6f7873ee0c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:23.010912 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-7448bd4e-a4ea-4de8-acc4-4bda3f868e3e None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:23.012562 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-7448bd4e-a4ea-4de8-acc4-4bda3f868e3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:23.041051 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-f051f926-c019-4472-b5b9-a6f7873ee0c3 None None] MFA Rules not processed for user `17d4eb55809b49039a6cca481552cef2`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:23.042529 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-f051f926-c019-4472-b5b9-a6f7873ee0c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:23.076417 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:23.091394 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1141/2279] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:54:23.096110 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7448bd4e-a4ea-4de8-acc4-4bda3f868e3e None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:23.111699 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7448bd4e-a4ea-4de8-acc4-4bda3f868e3e None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:23.118774 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10607295-6a16-402c-af7d-db66d435e408 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:23.119538 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10607295-6a16-402c-af7d-db66d435e408 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:23.120373 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10607295-6a16-402c-af7d-db66d435e408 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:23.120644 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10607295-6a16-402c-af7d-db66d435e408 admin admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:23.121483 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10607295-6a16-402c-af7d-db66d435e408 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:54:23.121918 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10607295-6a16-402c-af7d-db66d435e408 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:23.123908 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10607295-6a16-402c-af7d-db66d435e408 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:23.142240 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1142/2280] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 00:54:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:54:23.146908 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10607295-6a16-402c-af7d-db66d435e408 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:23.149110 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1139/2281] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:23 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:54:23.160683 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10607295-6a16-402c-af7d-db66d435e408 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:23.161462 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f051f926-c019-4472-b5b9-a6f7873ee0c3 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:23.163487 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-f051f926-c019-4472-b5b9-a6f7873ee0c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:23.168073 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c440c98-864b-41d5-b59e-45371f999c0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:23.168693 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c440c98-864b-41d5-b59e-45371f999c0a admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:23.169057 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c440c98-864b-41d5-b59e-45371f999c0a admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:23.169339 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c440c98-864b-41d5-b59e-45371f999c0a admin admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:23.170177 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c440c98-864b-41d5-b59e-45371f999c0a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:23.173390 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c440c98-864b-41d5-b59e-45371f999c0a admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:23.213008 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1143/2282] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:54:23 2022] POST /identity/v3/projects => generated 371 bytes in 67 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:54:23.214582 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c440c98-864b-41d5-b59e-45371f999c0a admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:23.227918 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f051f926-c019-4472-b5b9-a6f7873ee0c3 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:23.272221 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:23.286516 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c36edab8-e5f6-4a0a-a746-9f05710db14f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:23.287325 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36edab8-e5f6-4a0a-a746-9f05710db14f service cinder] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:23.287604 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36edab8-e5f6-4a0a-a746-9f05710db14f service cinder] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:23.287947 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36edab8-e5f6-4a0a-a746-9f05710db14f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:23.300610 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c440c98-864b-41d5-b59e-45371f999c0a admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:23.305860 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d858406-eecd-4535-8478-60118e625d3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:23.306518 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d858406-eecd-4535-8478-60118e625d3c admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:23.306766 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d858406-eecd-4535-8478-60118e625d3c admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:23.307181 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d858406-eecd-4535-8478-60118e625d3c admin admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:23.307899 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d858406-eecd-4535-8478-60118e625d3c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:23.309800 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d858406-eecd-4535-8478-60118e625d3c admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:23.334495 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1144/2283] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:54:23 2022] POST /identity/v3/users => generated 286 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:54:23.335637 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d858406-eecd-4535-8478-60118e625d3c admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:23.337261 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-0d858406-eecd-4535-8478-60118e625d3c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:23.350204 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d858406-eecd-4535-8478-60118e625d3c admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:23.352885 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c36edab8-e5f6-4a0a-a746-9f05710db14f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:23.353776 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f5ef820-f479-4aff-8882-57820e1be204 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:23.355003 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c36edab8-e5f6-4a0a-a746-9f05710db14f service cinder] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:23.355291 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f5ef820-f479-4aff-8882-57820e1be204 service cinder] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:23.355572 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f5ef820-f479-4aff-8882-57820e1be204 service cinder] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:23.355845 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f5ef820-f479-4aff-8882-57820e1be204 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:23.357589 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-5f5ef820-f479-4aff-8882-57820e1be204 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:23.364232 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1140/2284] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:54:23 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 203 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 11 00:54:23.365014 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c36edab8-e5f6-4a0a-a746-9f05710db14f service cinder] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:23.379809 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c36edab8-e5f6-4a0a-a746-9f05710db14f service cinder] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:23.386089 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-680ab83b-6749-4dcc-96db-c9902ace5116 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:23.386702 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-680ab83b-6749-4dcc-96db-c9902ace5116 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:23.387013 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-680ab83b-6749-4dcc-96db-c9902ace5116 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:23.387283 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-680ab83b-6749-4dcc-96db-c9902ace5116 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:23.387947 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-680ab83b-6749-4dcc-96db-c9902ace5116 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:23.389625 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-680ab83b-6749-4dcc-96db-c9902ace5116 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:23.400444 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1141/2285] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:54:23 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:54:23.404926 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-680ab83b-6749-4dcc-96db-c9902ace5116 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:23.415133 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-680ab83b-6749-4dcc-96db-c9902ace5116 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:23.421014 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48adc53d-2da8-435f-a9b7-42f73e71a670 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:23.421611 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48adc53d-2da8-435f-a9b7-42f73e71a670 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:23.421887 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48adc53d-2da8-435f-a9b7-42f73e71a670 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:23.422158 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48adc53d-2da8-435f-a9b7-42f73e71a670 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:23.422821 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48adc53d-2da8-435f-a9b7-42f73e71a670 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:23.424410 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48adc53d-2da8-435f-a9b7-42f73e71a670 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:23.430723 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f5ef820-f479-4aff-8882-57820e1be204 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:23.432836 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f5ef820-f479-4aff-8882-57820e1be204 service cinder] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:23.435456 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1142/2286] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:54:23 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:54:23.442896 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48adc53d-2da8-435f-a9b7-42f73e71a670 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:23.450822 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1145/2287] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:54:23 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:54:23.462668 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48adc53d-2da8-435f-a9b7-42f73e71a670 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:23.468837 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8dd87a-f90e-4dba-97fc-a343d959c853 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:23.469501 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8dd87a-f90e-4dba-97fc-a343d959c853 admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:23.469778 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8dd87a-f90e-4dba-97fc-a343d959c853 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:23.470053 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8dd87a-f90e-4dba-97fc-a343d959c853 admin admin] PATH_INFO: `/v3/projects/95a71d9b3f4646ee8eadc650dc73081a/users/e5bc790f99f74699b98ebc8ebca2bad5/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:23.486313 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:23.495277 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a8dd87a-f90e-4dba-97fc-a343d959c853 admin admin] RBAC: Authorizing `identity:create_grant(project_id=95a71d9b3f4646ee8eadc650dc73081a, user_id=e5bc790f99f74699b98ebc8ebca2bad5, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:23.498648 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a8dd87a-f90e-4dba-97fc-a343d959c853 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:23.525003 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1143/2288] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:54:23 2022] PUT /identity/v3/projects/95a71d9b3f4646ee8eadc650dc73081a/users/e5bc790f99f74699b98ebc8ebca2bad5/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 85 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:54:23.537778 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5186b4d5-cbf3-4f94-bed7-e86e1fbc2c86 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:23.538227 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5186b4d5-cbf3-4f94-bed7-e86e1fbc2c86 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:23.538610 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5186b4d5-cbf3-4f94-bed7-e86e1fbc2c86 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:23.587887 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-5186b4d5-cbf3-4f94-bed7-e86e1fbc2c86 None None] MFA Rules not processed for user `e5bc790f99f74699b98ebc8ebca2bad5`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:23.589865 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-5186b4d5-cbf3-4f94-bed7-e86e1fbc2c86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:23.711212 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1146/2289] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:23 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:54:23.722962 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8dd87a-f90e-4dba-97fc-a343d959c853 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:23.796387 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8dd87a-f90e-4dba-97fc-a343d959c853 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:23.800633 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97f60476-e3c1-4c48-8b01-fdf9384e5774 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:23.801278 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f60476-e3c1-4c48-8b01-fdf9384e5774 service cinder] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:23.801558 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f60476-e3c1-4c48-8b01-fdf9384e5774 service cinder] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:23.801827 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f60476-e3c1-4c48-8b01-fdf9384e5774 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:23.819119 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97f60476-e3c1-4c48-8b01-fdf9384e5774 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:23.820679 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97f60476-e3c1-4c48-8b01-fdf9384e5774 service cinder] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:23.832309 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1144/2290] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:54:23 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:54:23.861744 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5186b4d5-cbf3-4f94-bed7-e86e1fbc2c86 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:23.934875 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:23.938208 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5186b4d5-cbf3-4f94-bed7-e86e1fbc2c86 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:23.947217 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d1cd80-2e42-441c-8bac-6308911ff14d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:23.949108 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d1cd80-2e42-441c-8bac-6308911ff14d admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:23.949387 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d1cd80-2e42-441c-8bac-6308911ff14d admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:23.949659 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d1cd80-2e42-441c-8bac-6308911ff14d admin admin] PATH_INFO: `/v3/users/e5bc790f99f74699b98ebc8ebca2bad5` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:23.950892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1d1cd80-2e42-441c-8bac-6308911ff14d admin admin] RBAC: Authorizing `identity:delete_user(user_id=e5bc790f99f74699b98ebc8ebca2bad5)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:23.953401 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1d1cd80-2e42-441c-8bac-6308911ff14d admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:23.983775 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97f60476-e3c1-4c48-8b01-fdf9384e5774 service cinder] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:23.985506 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-97f60476-e3c1-4c48-8b01-fdf9384e5774 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:24.001610 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97f60476-e3c1-4c48-8b01-fdf9384e5774 service cinder] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:24.006270 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-905ce51e-aa3b-4069-ad35-dd1ee77f008a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:24.006911 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905ce51e-aa3b-4069-ad35-dd1ee77f008a admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:24.007234 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905ce51e-aa3b-4069-ad35-dd1ee77f008a admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:24.007509 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905ce51e-aa3b-4069-ad35-dd1ee77f008a admin admin] PATH_INFO: `/v3/users/ea94872269d14d05be54048753f683a1` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:24.008022 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-e1d1cd80-2e42-441c-8bac-6308911ff14d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e5bc790f99f74699b98ebc8ebca2bad5'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:24.008667 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-e1d1cd80-2e42-441c-8bac-6308911ff14d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e5bc790f99f74699b98ebc8ebca2bad5'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:24.008818 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-905ce51e-aa3b-4069-ad35-dd1ee77f008a admin admin] RBAC: Authorizing `identity:delete_user(user_id=ea94872269d14d05be54048753f683a1)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:24.009309 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-e1d1cd80-2e42-441c-8bac-6308911ff14d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e5bc790f99f74699b98ebc8ebca2bad5'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:24.011072 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-905ce51e-aa3b-4069-ad35-dd1ee77f008a admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:24.017123 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-e1d1cd80-2e42-441c-8bac-6308911ff14d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e5bc790f99f74699b98ebc8ebca2bad5'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:24.029725 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-e1d1cd80-2e42-441c-8bac-6308911ff14d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e5bc790f99f74699b98ebc8ebca2bad5'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:24.045359 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1147/2291] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:54:23 2022] DELETE /identity/v3/users/e5bc790f99f74699b98ebc8ebca2bad5 => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:54:24.049955 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce679c6c-1505-4ba4-b8cb-f4aa3a9391df None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:24.050267 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce679c6c-1505-4ba4-b8cb-f4aa3a9391df None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:24.050545 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce679c6c-1505-4ba4-b8cb-f4aa3a9391df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:24.060827 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-905ce51e-aa3b-4069-ad35-dd1ee77f008a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ea94872269d14d05be54048753f683a1'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:24.061481 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-905ce51e-aa3b-4069-ad35-dd1ee77f008a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ea94872269d14d05be54048753f683a1'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:24.062090 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-905ce51e-aa3b-4069-ad35-dd1ee77f008a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ea94872269d14d05be54048753f683a1'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:24.069545 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-905ce51e-aa3b-4069-ad35-dd1ee77f008a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ea94872269d14d05be54048753f683a1'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:24.070414 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-ce679c6c-1505-4ba4-b8cb-f4aa3a9391df None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:24.071639 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-ce679c6c-1505-4ba4-b8cb-f4aa3a9391df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:24.083441 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-905ce51e-aa3b-4069-ad35-dd1ee77f008a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ea94872269d14d05be54048753f683a1'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:24.097731 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1145/2292] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:54:23 2022] DELETE /identity/v3/users/ea94872269d14d05be54048753f683a1 => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:54:24.102756 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69342d51-53d9-4ea8-b687-75eb9750dbb2 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:24.103064 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69342d51-53d9-4ea8-b687-75eb9750dbb2 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:24.103312 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69342d51-53d9-4ea8-b687-75eb9750dbb2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:24.123242 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-69342d51-53d9-4ea8-b687-75eb9750dbb2 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:24.124452 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-69342d51-53d9-4ea8-b687-75eb9750dbb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:24.138661 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1148/2293] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:54:24.148398 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce679c6c-1505-4ba4-b8cb-f4aa3a9391df None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:24.149777 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-ce679c6c-1505-4ba4-b8cb-f4aa3a9391df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:24.200860 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1146/2294] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:54:24.210499 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69342d51-53d9-4ea8-b687-75eb9750dbb2 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:24.211149 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce679c6c-1505-4ba4-b8cb-f4aa3a9391df None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:24.219892 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:24.242888 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:24.266066 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7982e238-2414-4c0e-822e-f6e022eec416 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:24.266679 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7982e238-2414-4c0e-822e-f6e022eec416 service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:24.266984 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7982e238-2414-4c0e-822e-f6e022eec416 service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:24.267256 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7982e238-2414-4c0e-822e-f6e022eec416 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:24.270673 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69342d51-53d9-4ea8-b687-75eb9750dbb2 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:24.274262 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb376c2-3b0e-4ca1-bfcc-b37c2d9d81bd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:24.274893 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb376c2-3b0e-4ca1-bfcc-b37c2d9d81bd service neutron] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:24.275171 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb376c2-3b0e-4ca1-bfcc-b37c2d9d81bd service neutron] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:24.275437 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb376c2-3b0e-4ca1-bfcc-b37c2d9d81bd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:24.277358 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7982e238-2414-4c0e-822e-f6e022eec416 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:24.278802 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7982e238-2414-4c0e-822e-f6e022eec416 service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:24.286039 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eb376c2-3b0e-4ca1-bfcc-b37c2d9d81bd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:24.287491 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eb376c2-3b0e-4ca1-bfcc-b37c2d9d81bd service neutron] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:24.288774 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1149/2295] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:54:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 11 00:54:24.300951 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1147/2296] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:54:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 11 00:54:24.706072 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7982e238-2414-4c0e-822e-f6e022eec416 service neutron] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:24.707536 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-7982e238-2414-4c0e-822e-f6e022eec416 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:24.755085 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7982e238-2414-4c0e-822e-f6e022eec416 service neutron] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:24.768523 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f6ca190-2249-4b3c-ae69-8069e32837a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:24.769678 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6ca190-2249-4b3c-ae69-8069e32837a9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:24.770238 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6ca190-2249-4b3c-ae69-8069e32837a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:24.770852 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6ca190-2249-4b3c-ae69-8069e32837a9 admin admin] PATH_INFO: `/v3/projects/95a71d9b3f4646ee8eadc650dc73081a` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:24.773254 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f6ca190-2249-4b3c-ae69-8069e32837a9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=95a71d9b3f4646ee8eadc650dc73081a)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:24.777595 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f6ca190-2249-4b3c-ae69-8069e32837a9 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:24.830415 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb376c2-3b0e-4ca1-bfcc-b37c2d9d81bd service neutron] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:24.832845 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-9eb376c2-3b0e-4ca1-bfcc-b37c2d9d81bd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:24.867348 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb376c2-3b0e-4ca1-bfcc-b37c2d9d81bd service neutron] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:24.896971 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-6f6ca190-2249-4b3c-ae69-8069e32837a9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '95a71d9b3f4646ee8eadc650dc73081a', 'request_id': 'req-6f6ca190-2249-4b3c-ae69-8069e32837a9'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:24.909923 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-6f6ca190-2249-4b3c-ae69-8069e32837a9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '95a71d9b3f4646ee8eadc650dc73081a', 'request_id': 'req-6f6ca190-2249-4b3c-ae69-8069e32837a9'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:24.923680 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-6f6ca190-2249-4b3c-ae69-8069e32837a9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '95a71d9b3f4646ee8eadc650dc73081a', 'request_id': 'req-6f6ca190-2249-4b3c-ae69-8069e32837a9'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:24.943941 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-6f6ca190-2249-4b3c-ae69-8069e32837a9 admin admin] The token cache is being invalidate because project 95a71d9b3f4646ee8eadc650dc73081a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87924) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:54:24.943941 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-6f6ca190-2249-4b3c-ae69-8069e32837a9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:24.945135 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-6f6ca190-2249-4b3c-ae69-8069e32837a9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:24.955032 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1150/2297] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:54:24 2022] DELETE /identity/v3/projects/95a71d9b3f4646ee8eadc650dc73081a => generated 0 bytes in 244 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:54:25.058463 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-946c18df-f815-41ea-88c4-a4c3fca6c57f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:25.059163 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946c18df-f815-41ea-88c4-a4c3fca6c57f admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:25.059439 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946c18df-f815-41ea-88c4-a4c3fca6c57f admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:25.059725 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946c18df-f815-41ea-88c4-a4c3fca6c57f admin admin] PATH_INFO: `/v3/projects/bc794a63b6db4ba3b2f7293767c998fa` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:25.061068 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-946c18df-f815-41ea-88c4-a4c3fca6c57f admin admin] RBAC: Authorizing `identity:delete_project(project_id=bc794a63b6db4ba3b2f7293767c998fa)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:25.063133 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-946c18df-f815-41ea-88c4-a4c3fca6c57f admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:25.143015 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-946c18df-f815-41ea-88c4-a4c3fca6c57f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bc794a63b6db4ba3b2f7293767c998fa', 'request_id': 'req-946c18df-f815-41ea-88c4-a4c3fca6c57f'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:25.157547 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-946c18df-f815-41ea-88c4-a4c3fca6c57f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bc794a63b6db4ba3b2f7293767c998fa', 'request_id': 'req-946c18df-f815-41ea-88c4-a4c3fca6c57f'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:25.181038 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-946c18df-f815-41ea-88c4-a4c3fca6c57f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bc794a63b6db4ba3b2f7293767c998fa', 'request_id': 'req-946c18df-f815-41ea-88c4-a4c3fca6c57f'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:25.199943 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-946c18df-f815-41ea-88c4-a4c3fca6c57f admin admin] The token cache is being invalidate because project bc794a63b6db4ba3b2f7293767c998fa was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87923) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:54:25.200236 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-946c18df-f815-41ea-88c4-a4c3fca6c57f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:25.200887 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-946c18df-f815-41ea-88c4-a4c3fca6c57f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:25.202977 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1148/2298] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:54:24 2022] DELETE /identity/v3/projects/bc794a63b6db4ba3b2f7293767c998fa => generated 0 bytes in 374 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:54:27.669389 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f6ca190-2249-4b3c-ae69-8069e32837a9 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:27.670951 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-6f6ca190-2249-4b3c-ae69-8069e32837a9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:27.685498 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f6ca190-2249-4b3c-ae69-8069e32837a9 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:27.744770 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95866ef9-c40e-4c2a-9a2c-8b57d4aa02a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:27.745406 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95866ef9-c40e-4c2a-9a2c-8b57d4aa02a7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:27.745688 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95866ef9-c40e-4c2a-9a2c-8b57d4aa02a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:27.745959 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95866ef9-c40e-4c2a-9a2c-8b57d4aa02a7 admin admin] PATH_INFO: `/v3/users/0cb02c3d58444627913b6462fd1090d2` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:27.747133 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95866ef9-c40e-4c2a-9a2c-8b57d4aa02a7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0cb02c3d58444627913b6462fd1090d2)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:27.749078 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95866ef9-c40e-4c2a-9a2c-8b57d4aa02a7 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:27.811124 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-95866ef9-c40e-4c2a-9a2c-8b57d4aa02a7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0cb02c3d58444627913b6462fd1090d2'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:27.819453 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-95866ef9-c40e-4c2a-9a2c-8b57d4aa02a7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0cb02c3d58444627913b6462fd1090d2'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:27.831067 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-95866ef9-c40e-4c2a-9a2c-8b57d4aa02a7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0cb02c3d58444627913b6462fd1090d2'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:27.846557 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-95866ef9-c40e-4c2a-9a2c-8b57d4aa02a7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0cb02c3d58444627913b6462fd1090d2'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:27.852451 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:27.874576 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-95866ef9-c40e-4c2a-9a2c-8b57d4aa02a7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0cb02c3d58444627913b6462fd1090d2'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:27.889041 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1151/2299] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:54:27 2022] DELETE /identity/v3/users/0cb02c3d58444627913b6462fd1090d2 => generated 0 bytes in 220 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:54:27.893633 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4a8db1-4705-49cb-8cc3-8ee7ed5acc1c None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:27.894137 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4a8db1-4705-49cb-8cc3-8ee7ed5acc1c None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:27.894310 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4a8db1-4705-49cb-8cc3-8ee7ed5acc1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:27.914410 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-5d4a8db1-4705-49cb-8cc3-8ee7ed5acc1c None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:27.915711 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-5d4a8db1-4705-49cb-8cc3-8ee7ed5acc1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:27.953191 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:27.989076 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1149/2300] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:54:27.998176 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95866ef9-c40e-4c2a-9a2c-8b57d4aa02a7 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:28.000086 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-95866ef9-c40e-4c2a-9a2c-8b57d4aa02a7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:28.073872 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95866ef9-c40e-4c2a-9a2c-8b57d4aa02a7 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:28.083542 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87f64b39-4bf8-4b3a-b16c-24a6db29050b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:28.084605 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f64b39-4bf8-4b3a-b16c-24a6db29050b service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:28.085117 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f64b39-4bf8-4b3a-b16c-24a6db29050b service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:28.085395 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f64b39-4bf8-4b3a-b16c-24a6db29050b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:28.095620 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87f64b39-4bf8-4b3a-b16c-24a6db29050b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:28.097456 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87f64b39-4bf8-4b3a-b16c-24a6db29050b service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:28.127574 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1152/2301] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:54:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:54:28.415229 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d4a8db1-4705-49cb-8cc3-8ee7ed5acc1c None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:28.417444 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-5d4a8db1-4705-49cb-8cc3-8ee7ed5acc1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:28.430597 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d4a8db1-4705-49cb-8cc3-8ee7ed5acc1c None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:28.436072 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f810d54-c203-480d-bb1f-590b121f3b87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:28.436671 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f810d54-c203-480d-bb1f-590b121f3b87 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:28.436946 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f810d54-c203-480d-bb1f-590b121f3b87 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:28.437219 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f810d54-c203-480d-bb1f-590b121f3b87 admin admin] PATH_INFO: `/v3/projects/15d1340dbc59490ca4d9c56802b9cef6` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:28.438345 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f810d54-c203-480d-bb1f-590b121f3b87 admin admin] RBAC: Authorizing `identity:delete_project(project_id=15d1340dbc59490ca4d9c56802b9cef6)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:28.440347 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f810d54-c203-480d-bb1f-590b121f3b87 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:28.490563 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-8f810d54-c203-480d-bb1f-590b121f3b87 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '15d1340dbc59490ca4d9c56802b9cef6', 'request_id': 'req-8f810d54-c203-480d-bb1f-590b121f3b87'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:28.496991 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-8f810d54-c203-480d-bb1f-590b121f3b87 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '15d1340dbc59490ca4d9c56802b9cef6', 'request_id': 'req-8f810d54-c203-480d-bb1f-590b121f3b87'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:28.508437 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-8f810d54-c203-480d-bb1f-590b121f3b87 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '15d1340dbc59490ca4d9c56802b9cef6', 'request_id': 'req-8f810d54-c203-480d-bb1f-590b121f3b87'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:28.522011 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-8f810d54-c203-480d-bb1f-590b121f3b87 admin admin] The token cache is being invalidate because project 15d1340dbc59490ca4d9c56802b9cef6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87923) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:54:28.522302 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-8f810d54-c203-480d-bb1f-590b121f3b87 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:28.522963 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-8f810d54-c203-480d-bb1f-590b121f3b87 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:28.524442 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1150/2302] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:54:28 2022] DELETE /identity/v3/projects/15d1340dbc59490ca4d9c56802b9cef6 => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:54:28.528947 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87f64b39-4bf8-4b3a-b16c-24a6db29050b service neutron] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:28.538808 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87f64b39-4bf8-4b3a-b16c-24a6db29050b service neutron] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:28.605551 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b1e7b0-d69c-467f-ba82-1f0122f3eced admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:28.606183 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b1e7b0-d69c-467f-ba82-1f0122f3eced admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:28.606463 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b1e7b0-d69c-467f-ba82-1f0122f3eced admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:28.606736 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b1e7b0-d69c-467f-ba82-1f0122f3eced admin admin] PATH_INFO: `/v3/users/0f2e48fce4844476bca569acef4a4a62` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:28.607951 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b1e7b0-d69c-467f-ba82-1f0122f3eced admin admin] RBAC: Authorizing `identity:delete_user(user_id=0f2e48fce4844476bca569acef4a4a62)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:28.610139 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b1e7b0-d69c-467f-ba82-1f0122f3eced admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:28.659012 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-46b1e7b0-d69c-467f-ba82-1f0122f3eced admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0f2e48fce4844476bca569acef4a4a62'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:28.659814 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-46b1e7b0-d69c-467f-ba82-1f0122f3eced admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0f2e48fce4844476bca569acef4a4a62'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:28.660508 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-46b1e7b0-d69c-467f-ba82-1f0122f3eced admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0f2e48fce4844476bca569acef4a4a62'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:28.667552 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-46b1e7b0-d69c-467f-ba82-1f0122f3eced admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0f2e48fce4844476bca569acef4a4a62'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:28.681646 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-46b1e7b0-d69c-467f-ba82-1f0122f3eced admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0f2e48fce4844476bca569acef4a4a62'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:28.694331 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:DELETE=13,keystone:SELECT=85,keystone:INSERT=2 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:28.701107 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1153/2303] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:54:28 2022] DELETE /identity/v3/users/0f2e48fce4844476bca569acef4a4a62 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:54:29.001396 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f810d54-c203-480d-bb1f-590b121f3b87 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:29.002975 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-8f810d54-c203-480d-bb1f-590b121f3b87 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:29.017673 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f810d54-c203-480d-bb1f-590b121f3b87 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:29.085836 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca7129bd-af90-4f17-9e69-d390d9ce9aa6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:29.086615 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7129bd-af90-4f17-9e69-d390d9ce9aa6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:29.089879 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7129bd-af90-4f17-9e69-d390d9ce9aa6 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:29.089879 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7129bd-af90-4f17-9e69-d390d9ce9aa6 admin admin] PATH_INFO: `/v3/projects/dca34de5347f4f1d8ad840e8c4f81af1` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:29.089879 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca7129bd-af90-4f17-9e69-d390d9ce9aa6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=dca34de5347f4f1d8ad840e8c4f81af1)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:29.091225 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca7129bd-af90-4f17-9e69-d390d9ce9aa6 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:29.146890 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-ca7129bd-af90-4f17-9e69-d390d9ce9aa6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dca34de5347f4f1d8ad840e8c4f81af1', 'request_id': 'req-ca7129bd-af90-4f17-9e69-d390d9ce9aa6'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:29.153347 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-ca7129bd-af90-4f17-9e69-d390d9ce9aa6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dca34de5347f4f1d8ad840e8c4f81af1', 'request_id': 'req-ca7129bd-af90-4f17-9e69-d390d9ce9aa6'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:29.158932 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-ca7129bd-af90-4f17-9e69-d390d9ce9aa6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dca34de5347f4f1d8ad840e8c4f81af1', 'request_id': 'req-ca7129bd-af90-4f17-9e69-d390d9ce9aa6'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:29.161649 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:29.175388 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-ca7129bd-af90-4f17-9e69-d390d9ce9aa6 admin admin] The token cache is being invalidate because project dca34de5347f4f1d8ad840e8c4f81af1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87923) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:54:29.175736 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-ca7129bd-af90-4f17-9e69-d390d9ce9aa6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:29.176370 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-ca7129bd-af90-4f17-9e69-d390d9ce9aa6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:29.177960 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1151/2304] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:54:29 2022] DELETE /identity/v3/projects/dca34de5347f4f1d8ad840e8c4f81af1 => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:54:29.182489 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b1e7b0-d69c-467f-ba82-1f0122f3eced admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:29.184257 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-46b1e7b0-d69c-467f-ba82-1f0122f3eced admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:29.197489 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b1e7b0-d69c-467f-ba82-1f0122f3eced admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:29.255196 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1916fa5a-cef9-43e2-af40-9accd8fb99b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:29.255834 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1916fa5a-cef9-43e2-af40-9accd8fb99b5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:29.256109 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1916fa5a-cef9-43e2-af40-9accd8fb99b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:29.256376 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1916fa5a-cef9-43e2-af40-9accd8fb99b5 admin admin] PATH_INFO: `/v3/users/501c024d392143d6b3ea7448916f470f` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:29.257503 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1916fa5a-cef9-43e2-af40-9accd8fb99b5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=501c024d392143d6b3ea7448916f470f)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:29.259535 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1916fa5a-cef9-43e2-af40-9accd8fb99b5 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:29.311541 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-1916fa5a-cef9-43e2-af40-9accd8fb99b5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '501c024d392143d6b3ea7448916f470f'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:29.312183 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-1916fa5a-cef9-43e2-af40-9accd8fb99b5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '501c024d392143d6b3ea7448916f470f'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:29.312777 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-1916fa5a-cef9-43e2-af40-9accd8fb99b5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '501c024d392143d6b3ea7448916f470f'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:29.319916 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-1916fa5a-cef9-43e2-af40-9accd8fb99b5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '501c024d392143d6b3ea7448916f470f'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:29.332261 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-1916fa5a-cef9-43e2-af40-9accd8fb99b5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '501c024d392143d6b3ea7448916f470f'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:29.346830 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1154/2305] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:54:29 2022] DELETE /identity/v3/users/501c024d392143d6b3ea7448916f470f => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:54:29.627629 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca7129bd-af90-4f17-9e69-d390d9ce9aa6 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:29.629128 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-ca7129bd-af90-4f17-9e69-d390d9ce9aa6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:29.645134 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca7129bd-af90-4f17-9e69-d390d9ce9aa6 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:29.706051 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bfb6bd9-a1b8-4ec7-b4e3-b6825317e26c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:29.706679 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bfb6bd9-a1b8-4ec7-b4e3-b6825317e26c admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:29.707096 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bfb6bd9-a1b8-4ec7-b4e3-b6825317e26c admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:29.707375 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bfb6bd9-a1b8-4ec7-b4e3-b6825317e26c admin admin] PATH_INFO: `/v3/projects/61aa2fdb893140b3a0fc3ffbe223970f` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:29.708517 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bfb6bd9-a1b8-4ec7-b4e3-b6825317e26c admin admin] RBAC: Authorizing `identity:delete_project(project_id=61aa2fdb893140b3a0fc3ffbe223970f)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:29.710419 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bfb6bd9-a1b8-4ec7-b4e3-b6825317e26c admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:29.761983 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-2bfb6bd9-a1b8-4ec7-b4e3-b6825317e26c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '61aa2fdb893140b3a0fc3ffbe223970f', 'request_id': 'req-2bfb6bd9-a1b8-4ec7-b4e3-b6825317e26c'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:29.768441 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-2bfb6bd9-a1b8-4ec7-b4e3-b6825317e26c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '61aa2fdb893140b3a0fc3ffbe223970f', 'request_id': 'req-2bfb6bd9-a1b8-4ec7-b4e3-b6825317e26c'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:29.774907 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-2bfb6bd9-a1b8-4ec7-b4e3-b6825317e26c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '61aa2fdb893140b3a0fc3ffbe223970f', 'request_id': 'req-2bfb6bd9-a1b8-4ec7-b4e3-b6825317e26c'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:29.788601 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-2bfb6bd9-a1b8-4ec7-b4e3-b6825317e26c admin admin] The token cache is being invalidate because project 61aa2fdb893140b3a0fc3ffbe223970f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87923) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:54:29.788889 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-2bfb6bd9-a1b8-4ec7-b4e3-b6825317e26c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:29.789625 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-2bfb6bd9-a1b8-4ec7-b4e3-b6825317e26c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:29.791253 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1152/2306] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:54:29 2022] DELETE /identity/v3/projects/61aa2fdb893140b3a0fc3ffbe223970f => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:54:29.797010 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5799cd4f-3d04-4314-8cc2-e4452f004c23 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:29.797506 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5799cd4f-3d04-4314-8cc2-e4452f004c23 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:29.797702 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5799cd4f-3d04-4314-8cc2-e4452f004c23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:29.817957 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-5799cd4f-3d04-4314-8cc2-e4452f004c23 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:29.819205 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-5799cd4f-3d04-4314-8cc2-e4452f004c23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:29.880672 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1155/2307] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:54:29.889246 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf45a3a-f5fb-4e69-9d35-4cbe7bf07f5d None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:29.889533 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf45a3a-f5fb-4e69-9d35-4cbe7bf07f5d None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:29.891169 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf45a3a-f5fb-4e69-9d35-4cbe7bf07f5d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:29.917550 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-daf45a3a-f5fb-4e69-9d35-4cbe7bf07f5d None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:29.918840 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-daf45a3a-f5fb-4e69-9d35-4cbe7bf07f5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:29.928726 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1153/2308] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:54:29.934304 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5799cd4f-3d04-4314-8cc2-e4452f004c23 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:29.945665 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5799cd4f-3d04-4314-8cc2-e4452f004c23 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:29.951971 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6cefd1f-747f-47c8-8227-07f6ea5e6e9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:29.953231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6cefd1f-747f-47c8-8227-07f6ea5e6e9a admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:29.953820 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6cefd1f-747f-47c8-8227-07f6ea5e6e9a admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:29.954568 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6cefd1f-747f-47c8-8227-07f6ea5e6e9a admin admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:29.955539 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6cefd1f-747f-47c8-8227-07f6ea5e6e9a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:54:29.956031 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6cefd1f-747f-47c8-8227-07f6ea5e6e9a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:29.957889 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6cefd1f-747f-47c8-8227-07f6ea5e6e9a admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:29.986602 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1156/2309] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 00:54:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:54:29.988226 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf45a3a-f5fb-4e69-9d35-4cbe7bf07f5d None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:29.999027 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf45a3a-f5fb-4e69-9d35-4cbe7bf07f5d None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:30.004111 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b682d9c-a306-45e3-933b-3a02b0e23d66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:30.004730 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b682d9c-a306-45e3-933b-3a02b0e23d66 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:30.005007 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b682d9c-a306-45e3-933b-3a02b0e23d66 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:30.005278 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b682d9c-a306-45e3-933b-3a02b0e23d66 admin admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:30.005966 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b682d9c-a306-45e3-933b-3a02b0e23d66 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:30.008170 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b682d9c-a306-45e3-933b-3a02b0e23d66 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:30.042029 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1154/2310] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:54:29 2022] POST /identity/v3/projects => generated 383 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:54:30.045676 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6cefd1f-747f-47c8-8227-07f6ea5e6e9a admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:30.057861 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6cefd1f-747f-47c8-8227-07f6ea5e6e9a admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:30.067208 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:30.132581 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b5ae83-e949-414a-8104-060d72dcd45d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:30.133199 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b5ae83-e949-414a-8104-060d72dcd45d admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:30.133475 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b5ae83-e949-414a-8104-060d72dcd45d admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:30.133746 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b5ae83-e949-414a-8104-060d72dcd45d admin admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:30.134408 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b5ae83-e949-414a-8104-060d72dcd45d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:30.136510 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b5ae83-e949-414a-8104-060d72dcd45d admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:30.160826 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1157/2311] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:54:30 2022] POST /identity/v3/users => generated 292 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:54:30.165442 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b682d9c-a306-45e3-933b-3a02b0e23d66 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:30.185693 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b682d9c-a306-45e3-933b-3a02b0e23d66 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:30.190697 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db62356-7689-4c1a-85f2-dcd1f26343e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:30.191529 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db62356-7689-4c1a-85f2-dcd1f26343e5 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:30.191808 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db62356-7689-4c1a-85f2-dcd1f26343e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:30.192083 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db62356-7689-4c1a-85f2-dcd1f26343e5 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:30.192760 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3db62356-7689-4c1a-85f2-dcd1f26343e5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:30.194353 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3db62356-7689-4c1a-85f2-dcd1f26343e5 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:30.205773 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1155/2312] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:54:30 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 11 00:54:30.209810 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b5ae83-e949-414a-8104-060d72dcd45d admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:30.220363 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b5ae83-e949-414a-8104-060d72dcd45d admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:30.224930 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a99643a9-5885-451a-82c9-274f348e0310 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:30.227729 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99643a9-5885-451a-82c9-274f348e0310 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:30.227729 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99643a9-5885-451a-82c9-274f348e0310 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:30.227729 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99643a9-5885-451a-82c9-274f348e0310 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:30.227729 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a99643a9-5885-451a-82c9-274f348e0310 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:30.228285 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a99643a9-5885-451a-82c9-274f348e0310 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:30.238763 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1158/2313] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:54:30 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:54:30.243158 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db62356-7689-4c1a-85f2-dcd1f26343e5 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:30.255756 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db62356-7689-4c1a-85f2-dcd1f26343e5 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:30.261515 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53e449de-ab02-4a1a-8bc9-a35311b1a830 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:30.262345 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e449de-ab02-4a1a-8bc9-a35311b1a830 admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:30.262518 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e449de-ab02-4a1a-8bc9-a35311b1a830 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:30.262898 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e449de-ab02-4a1a-8bc9-a35311b1a830 admin admin] PATH_INFO: `/v3/projects/b8fee68ea94a4f22b408526912ac2fa3/users/cc8f01a8f81d4b27a89ef399c9d574d0/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:30.282076 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53e449de-ab02-4a1a-8bc9-a35311b1a830 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b8fee68ea94a4f22b408526912ac2fa3, user_id=cc8f01a8f81d4b27a89ef399c9d574d0, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:30.285037 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53e449de-ab02-4a1a-8bc9-a35311b1a830 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:30.299931 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1156/2314] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:54:30 2022] PUT /identity/v3/projects/b8fee68ea94a4f22b408526912ac2fa3/users/cc8f01a8f81d4b27a89ef399c9d574d0/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:54:30.312221 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9618de94-353d-48e3-a149-bd65e23dfacd None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:30.312506 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9618de94-353d-48e3-a149-bd65e23dfacd None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:30.312832 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9618de94-353d-48e3-a149-bd65e23dfacd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:30.341796 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-9618de94-353d-48e3-a149-bd65e23dfacd None None] MFA Rules not processed for user `cc8f01a8f81d4b27a89ef399c9d574d0`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:30.343047 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-9618de94-353d-48e3-a149-bd65e23dfacd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:30.413174 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1159/2315] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:30 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:54:30.425277 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53e449de-ab02-4a1a-8bc9-a35311b1a830 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:30.427007 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-53e449de-ab02-4a1a-8bc9-a35311b1a830 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:30.449511 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:DELETE=8,keystone:SELECT=89,keystone:INSERT=3 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:30.487693 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53e449de-ab02-4a1a-8bc9-a35311b1a830 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:30.491341 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fedc2804-b3cf-4d88-afa9-b2c9a9fedb69 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:30.491946 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fedc2804-b3cf-4d88-afa9-b2c9a9fedb69 service cinder] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:30.492223 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fedc2804-b3cf-4d88-afa9-b2c9a9fedb69 service cinder] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:30.492495 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fedc2804-b3cf-4d88-afa9-b2c9a9fedb69 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:30.501802 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fedc2804-b3cf-4d88-afa9-b2c9a9fedb69 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:30.503382 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fedc2804-b3cf-4d88-afa9-b2c9a9fedb69 service cinder] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:30.513043 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1157/2316] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:54:30 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:54:33.081840 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9618de94-353d-48e3-a149-bd65e23dfacd None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:33.151310 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9618de94-353d-48e3-a149-bd65e23dfacd None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:33.155844 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a095c4ff-016d-43f4-843f-bf11f5991c9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:33.156468 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a095c4ff-016d-43f4-843f-bf11f5991c9e admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:33.156745 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a095c4ff-016d-43f4-843f-bf11f5991c9e admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:33.157013 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a095c4ff-016d-43f4-843f-bf11f5991c9e admin admin] PATH_INFO: `/v3/users/cc8f01a8f81d4b27a89ef399c9d574d0` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:33.158132 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a095c4ff-016d-43f4-843f-bf11f5991c9e admin admin] RBAC: Authorizing `identity:delete_user(user_id=cc8f01a8f81d4b27a89ef399c9d574d0)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:33.160357 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a095c4ff-016d-43f4-843f-bf11f5991c9e admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:33.175951 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=96,keystone:INSERT=3,keystone:DELETE=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:33.206877 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-a095c4ff-016d-43f4-843f-bf11f5991c9e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cc8f01a8f81d4b27a89ef399c9d574d0'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:33.207521 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-a095c4ff-016d-43f4-843f-bf11f5991c9e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cc8f01a8f81d4b27a89ef399c9d574d0'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:33.208114 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-a095c4ff-016d-43f4-843f-bf11f5991c9e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cc8f01a8f81d4b27a89ef399c9d574d0'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:33.215268 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-a095c4ff-016d-43f4-843f-bf11f5991c9e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cc8f01a8f81d4b27a89ef399c9d574d0'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:33.227490 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-a095c4ff-016d-43f4-843f-bf11f5991c9e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cc8f01a8f81d4b27a89ef399c9d574d0'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:33.240594 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1160/2317] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:54:33 2022] DELETE /identity/v3/users/cc8f01a8f81d4b27a89ef399c9d574d0 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:54:33.245651 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3371348-2528-49eb-b4e2-1384c61b15b8 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:33.246134 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3371348-2528-49eb-b4e2-1384c61b15b8 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:33.246227 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3371348-2528-49eb-b4e2-1384c61b15b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:33.265383 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-e3371348-2528-49eb-b4e2-1384c61b15b8 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:33.266569 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-e3371348-2528-49eb-b4e2-1384c61b15b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:33.325286 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1158/2318] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:54:33.334922 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a095c4ff-016d-43f4-843f-bf11f5991c9e admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:33.336325 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-a095c4ff-016d-43f4-843f-bf11f5991c9e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:33.394863 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a095c4ff-016d-43f4-843f-bf11f5991c9e admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:33.398525 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f0b77fd-a90e-406f-a520-04fb313f14fc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:33.399165 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0b77fd-a90e-406f-a520-04fb313f14fc service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:33.399445 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0b77fd-a90e-406f-a520-04fb313f14fc service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:33.399717 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0b77fd-a90e-406f-a520-04fb313f14fc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:33.408957 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f0b77fd-a90e-406f-a520-04fb313f14fc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:33.410635 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f0b77fd-a90e-406f-a520-04fb313f14fc service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:33.420671 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1161/2319] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:54:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:54:33.704241 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3371348-2528-49eb-b4e2-1384c61b15b8 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:33.707806 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-e3371348-2528-49eb-b4e2-1384c61b15b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:33.724932 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3371348-2528-49eb-b4e2-1384c61b15b8 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:33.730132 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5e69526-17d6-4290-9558-89a38a969d25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:33.730739 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e69526-17d6-4290-9558-89a38a969d25 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:33.730991 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e69526-17d6-4290-9558-89a38a969d25 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:33.731506 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e69526-17d6-4290-9558-89a38a969d25 admin admin] PATH_INFO: `/v3/projects/b8fee68ea94a4f22b408526912ac2fa3` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:33.732646 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5e69526-17d6-4290-9558-89a38a969d25 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b8fee68ea94a4f22b408526912ac2fa3)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:33.734681 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5e69526-17d6-4290-9558-89a38a969d25 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:33.791694 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-a5e69526-17d6-4290-9558-89a38a969d25 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b8fee68ea94a4f22b408526912ac2fa3', 'request_id': 'req-a5e69526-17d6-4290-9558-89a38a969d25'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:33.801031 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-a5e69526-17d6-4290-9558-89a38a969d25 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b8fee68ea94a4f22b408526912ac2fa3', 'request_id': 'req-a5e69526-17d6-4290-9558-89a38a969d25'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:33.807674 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-a5e69526-17d6-4290-9558-89a38a969d25 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b8fee68ea94a4f22b408526912ac2fa3', 'request_id': 'req-a5e69526-17d6-4290-9558-89a38a969d25'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:33.821357 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-a5e69526-17d6-4290-9558-89a38a969d25 admin admin] The token cache is being invalidate because project b8fee68ea94a4f22b408526912ac2fa3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87923) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:54:33.821645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-a5e69526-17d6-4290-9558-89a38a969d25 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:33.822460 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-a5e69526-17d6-4290-9558-89a38a969d25 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:33.824567 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1159/2320] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:54:33 2022] DELETE /identity/v3/projects/b8fee68ea94a4f22b408526912ac2fa3 => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:54:33.829286 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0348e249-5839-4c63-8ab4-d82ff2103bf1 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:33.829575 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0348e249-5839-4c63-8ab4-d82ff2103bf1 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:33.829852 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0348e249-5839-4c63-8ab4-d82ff2103bf1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:33.854388 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-0348e249-5839-4c63-8ab4-d82ff2103bf1 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:33.855392 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-0348e249-5839-4c63-8ab4-d82ff2103bf1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:33.888769 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5e69526-17d6-4290-9558-89a38a969d25 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:33.890381 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-a5e69526-17d6-4290-9558-89a38a969d25 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:33.954183 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1162/2321] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:54:33.956064 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:33.962192 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c17b88-7e15-4531-8237-05f10610ece7 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:33.962472 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c17b88-7e15-4531-8237-05f10610ece7 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:33.962782 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c17b88-7e15-4531-8237-05f10610ece7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:33.980737 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5e69526-17d6-4290-9558-89a38a969d25 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:33.984867 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-92c17b88-7e15-4531-8237-05f10610ece7 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:33.985968 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74e33471-1dae-4d16-a808-f1e671829195 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:33.986284 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-92c17b88-7e15-4531-8237-05f10610ece7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:33.986426 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e33471-1dae-4d16-a808-f1e671829195 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:33.986680 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e33471-1dae-4d16-a808-f1e671829195 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:33.987062 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e33471-1dae-4d16-a808-f1e671829195 admin admin] PATH_INFO: `/v3/users/17d4eb55809b49039a6cca481552cef2` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:33.988642 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74e33471-1dae-4d16-a808-f1e671829195 admin admin] RBAC: Authorizing `identity:delete_user(user_id=17d4eb55809b49039a6cca481552cef2)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:33.991495 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74e33471-1dae-4d16-a808-f1e671829195 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:34.000526 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1163/2322] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:54:34.007331 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c17b88-7e15-4531-8237-05f10610ece7 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:34.019280 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c17b88-7e15-4531-8237-05f10610ece7 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:34.024463 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af07f1be-19b0-4196-9937-50955de8766d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:34.025224 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af07f1be-19b0-4196-9937-50955de8766d admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:34.025627 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af07f1be-19b0-4196-9937-50955de8766d admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:34.026178 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af07f1be-19b0-4196-9937-50955de8766d admin admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:34.026976 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af07f1be-19b0-4196-9937-50955de8766d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:54:34.027350 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af07f1be-19b0-4196-9937-50955de8766d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:34.028962 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af07f1be-19b0-4196-9937-50955de8766d admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:34.042913 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1164/2323] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 00:54:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:54:34.047622 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af07f1be-19b0-4196-9937-50955de8766d admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:34.052177 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-74e33471-1dae-4d16-a808-f1e671829195 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '17d4eb55809b49039a6cca481552cef2'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:34.053542 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-74e33471-1dae-4d16-a808-f1e671829195 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '17d4eb55809b49039a6cca481552cef2'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:34.054601 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-74e33471-1dae-4d16-a808-f1e671829195 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '17d4eb55809b49039a6cca481552cef2'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:34.060852 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af07f1be-19b0-4196-9937-50955de8766d admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:34.065751 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b626c950-03ba-47a3-8c79-d5563900d2ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:34.066358 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b626c950-03ba-47a3-8c79-d5563900d2ea admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:34.066641 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b626c950-03ba-47a3-8c79-d5563900d2ea admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:34.066964 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b626c950-03ba-47a3-8c79-d5563900d2ea admin admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:34.067198 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-74e33471-1dae-4d16-a808-f1e671829195 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '17d4eb55809b49039a6cca481552cef2'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:34.067612 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b626c950-03ba-47a3-8c79-d5563900d2ea admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:34.069512 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b626c950-03ba-47a3-8c79-d5563900d2ea admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:34.086949 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-74e33471-1dae-4d16-a808-f1e671829195 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '17d4eb55809b49039a6cca481552cef2'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:34.089300 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:DELETE=9,keystone:SELECT=89,keystone:INSERT=2 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:34.108457 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1160/2324] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:54:33 2022] DELETE /identity/v3/users/17d4eb55809b49039a6cca481552cef2 => generated 0 bytes in 220 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:54:34.113803 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5dd0d78-58a9-4829-9948-d2de6606856d None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:34.114427 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5dd0d78-58a9-4829-9948-d2de6606856d None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:34.114549 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5dd0d78-58a9-4829-9948-d2de6606856d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:34.116558 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1165/2325] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:54:34 2022] POST /identity/v3/projects => generated 373 bytes in 69 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:54:34.121821 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b626c950-03ba-47a3-8c79-d5563900d2ea admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:34.123529 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-b626c950-03ba-47a3-8c79-d5563900d2ea admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:34.137089 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-c5dd0d78-58a9-4829-9948-d2de6606856d None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:34.138064 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-c5dd0d78-58a9-4829-9948-d2de6606856d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:34.197665 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b626c950-03ba-47a3-8c79-d5563900d2ea admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:34.203528 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f8132c2-80d8-4a61-89e8-23a9c9dd1fe6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:34.204215 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8132c2-80d8-4a61-89e8-23a9c9dd1fe6 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:34.204561 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8132c2-80d8-4a61-89e8-23a9c9dd1fe6 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:34.204899 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8132c2-80d8-4a61-89e8-23a9c9dd1fe6 admin admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:34.205600 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f8132c2-80d8-4a61-89e8-23a9c9dd1fe6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:34.207754 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f8132c2-80d8-4a61-89e8-23a9c9dd1fe6 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:34.210928 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1161/2326] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:54:34.225489 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5dd0d78-58a9-4829-9948-d2de6606856d None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:34.227509 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-c5dd0d78-58a9-4829-9948-d2de6606856d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:34.234339 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1166/2327] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:54:34 2022] POST /identity/v3/users => generated 287 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:54:34.239461 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f8132c2-80d8-4a61-89e8-23a9c9dd1fe6 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:34.254078 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f8132c2-80d8-4a61-89e8-23a9c9dd1fe6 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:34.261308 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33c944b-d1d5-4e46-b469-083e716d5d18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:34.261936 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33c944b-d1d5-4e46-b469-083e716d5d18 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:34.262238 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33c944b-d1d5-4e46-b469-083e716d5d18 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:34.262524 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33c944b-d1d5-4e46-b469-083e716d5d18 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:34.263481 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f33c944b-d1d5-4e46-b469-083e716d5d18 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:34.265559 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f33c944b-d1d5-4e46-b469-083e716d5d18 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:34.280737 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1167/2328] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:54:34 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 11 00:54:34.285138 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33c944b-d1d5-4e46-b469-083e716d5d18 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:34.298666 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5dd0d78-58a9-4829-9948-d2de6606856d None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:34.299276 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33c944b-d1d5-4e46-b469-083e716d5d18 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:34.302680 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d380e7b4-ec8d-4618-b907-56682d3d756d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:34.303395 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d380e7b4-ec8d-4618-b907-56682d3d756d service neutron] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:34.303675 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d380e7b4-ec8d-4618-b907-56682d3d756d service neutron] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:34.303960 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d380e7b4-ec8d-4618-b907-56682d3d756d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:34.305333 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33c5145b-e878-479a-ba64-3934d94b4451 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:34.306243 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c5145b-e878-479a-ba64-3934d94b4451 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:34.306880 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c5145b-e878-479a-ba64-3934d94b4451 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:34.307435 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c5145b-e878-479a-ba64-3934d94b4451 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:34.308440 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33c5145b-e878-479a-ba64-3934d94b4451 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:34.311059 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33c5145b-e878-479a-ba64-3934d94b4451 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:34.313972 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d380e7b4-ec8d-4618-b907-56682d3d756d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:34.315866 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d380e7b4-ec8d-4618-b907-56682d3d756d service neutron] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:34.327893 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1162/2329] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:54:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 11 00:54:34.332602 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1168/2330] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:54:34 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:54:34.333813 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d380e7b4-ec8d-4618-b907-56682d3d756d service neutron] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:34.344576 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d380e7b4-ec8d-4618-b907-56682d3d756d service neutron] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:34.349512 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f19dca3d-7de0-4d32-a467-273e4b0fffcb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:34.350221 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19dca3d-7de0-4d32-a467-273e4b0fffcb admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:34.350524 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19dca3d-7de0-4d32-a467-273e4b0fffcb admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:34.350844 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19dca3d-7de0-4d32-a467-273e4b0fffcb admin admin] PATH_INFO: `/v3/projects/d55d41063ec34fb6921c367025fa9af9/users/967e0ef4e6364804a58d0d59c22cd6a8/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:34.354494 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:34.370960 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f19dca3d-7de0-4d32-a467-273e4b0fffcb admin admin] RBAC: Authorizing `identity:create_grant(project_id=d55d41063ec34fb6921c367025fa9af9, user_id=967e0ef4e6364804a58d0d59c22cd6a8, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:34.373618 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f19dca3d-7de0-4d32-a467-273e4b0fffcb admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:34.394849 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1163/2331] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:54:34 2022] PUT /identity/v3/projects/d55d41063ec34fb6921c367025fa9af9/users/967e0ef4e6364804a58d0d59c22cd6a8/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:54:34.399748 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d436639a-8a71-407c-b9ff-4cf23b0eea7b None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:34.400036 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d436639a-8a71-407c-b9ff-4cf23b0eea7b None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:34.400874 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d436639a-8a71-407c-b9ff-4cf23b0eea7b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:34.425315 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-d436639a-8a71-407c-b9ff-4cf23b0eea7b None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:34.426532 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-d436639a-8a71-407c-b9ff-4cf23b0eea7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:34.517467 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1169/2332] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:54:34.533267 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f19dca3d-7de0-4d32-a467-273e4b0fffcb admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:34.607462 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f19dca3d-7de0-4d32-a467-273e4b0fffcb admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:34.611601 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58f4e8de-35a1-49c9-80cd-ed27ce93fd2b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:34.612384 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f4e8de-35a1-49c9-80cd-ed27ce93fd2b service neutron] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:34.612655 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f4e8de-35a1-49c9-80cd-ed27ce93fd2b service neutron] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:34.612923 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f4e8de-35a1-49c9-80cd-ed27ce93fd2b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:34.623847 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58f4e8de-35a1-49c9-80cd-ed27ce93fd2b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:34.625497 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58f4e8de-35a1-49c9-80cd-ed27ce93fd2b service neutron] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:34.639118 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1164/2333] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:54:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:54:34.665884 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d436639a-8a71-407c-b9ff-4cf23b0eea7b None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:34.667874 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-d436639a-8a71-407c-b9ff-4cf23b0eea7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:34.691954 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d436639a-8a71-407c-b9ff-4cf23b0eea7b None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:34.708852 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0bdca68-084f-4a7b-bf1b-ca2dfe2b5d2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:34.709223 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0bdca68-084f-4a7b-bf1b-ca2dfe2b5d2a admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:34.709511 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0bdca68-084f-4a7b-bf1b-ca2dfe2b5d2a admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:34.710017 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0bdca68-084f-4a7b-bf1b-ca2dfe2b5d2a admin admin] PATH_INFO: `/v3/projects/cdf84ece5642437a9b28d272e9ff5df2` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:34.711522 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0bdca68-084f-4a7b-bf1b-ca2dfe2b5d2a admin admin] RBAC: Authorizing `identity:delete_project(project_id=cdf84ece5642437a9b28d272e9ff5df2)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:34.713394 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0bdca68-084f-4a7b-bf1b-ca2dfe2b5d2a admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:34.768932 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-d0bdca68-084f-4a7b-bf1b-ca2dfe2b5d2a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cdf84ece5642437a9b28d272e9ff5df2', 'request_id': 'req-d0bdca68-084f-4a7b-bf1b-ca2dfe2b5d2a'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:34.777143 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-d0bdca68-084f-4a7b-bf1b-ca2dfe2b5d2a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cdf84ece5642437a9b28d272e9ff5df2', 'request_id': 'req-d0bdca68-084f-4a7b-bf1b-ca2dfe2b5d2a'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:34.782009 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=91,keystone:INSERT=4,keystone:DELETE=5 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:34.786421 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-d0bdca68-084f-4a7b-bf1b-ca2dfe2b5d2a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cdf84ece5642437a9b28d272e9ff5df2', 'request_id': 'req-d0bdca68-084f-4a7b-bf1b-ca2dfe2b5d2a'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:34.799562 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-d0bdca68-084f-4a7b-bf1b-ca2dfe2b5d2a admin admin] The token cache is being invalidate because project cdf84ece5642437a9b28d272e9ff5df2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87924) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:54:34.801515 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-d0bdca68-084f-4a7b-bf1b-ca2dfe2b5d2a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:34.802999 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-d0bdca68-084f-4a7b-bf1b-ca2dfe2b5d2a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:34.804781 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1170/2334] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:54:34 2022] DELETE /identity/v3/projects/cdf84ece5642437a9b28d272e9ff5df2 => generated 0 bytes in 139 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:54:34.810157 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a14c0d-4d33-4f65-b2ee-31f55d578f3d None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:34.810581 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a14c0d-4d33-4f65-b2ee-31f55d578f3d None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:34.810973 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a14c0d-4d33-4f65-b2ee-31f55d578f3d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:34.836817 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-66a14c0d-4d33-4f65-b2ee-31f55d578f3d None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:34.838839 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-66a14c0d-4d33-4f65-b2ee-31f55d578f3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:34.918925 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1165/2335] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:54:34.928014 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf28b35-fc43-4a7a-b76b-f4ec1678f49f None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:34.928670 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf28b35-fc43-4a7a-b76b-f4ec1678f49f None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:34.929954 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf28b35-fc43-4a7a-b76b-f4ec1678f49f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:34.963667 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-dbf28b35-fc43-4a7a-b76b-f4ec1678f49f None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:34.965152 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-dbf28b35-fc43-4a7a-b76b-f4ec1678f49f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:34.974904 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1171/2336] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 49 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:54:34.980084 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66a14c0d-4d33-4f65-b2ee-31f55d578f3d None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:34.991275 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66a14c0d-4d33-4f65-b2ee-31f55d578f3d None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:34.997019 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92377756-42e6-42c6-8ed7-93eacd674a0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:34.997811 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92377756-42e6-42c6-8ed7-93eacd674a0e admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:34.998087 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92377756-42e6-42c6-8ed7-93eacd674a0e admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:34.998359 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92377756-42e6-42c6-8ed7-93eacd674a0e admin admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:34.999062 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92377756-42e6-42c6-8ed7-93eacd674a0e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:54:34.999368 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92377756-42e6-42c6-8ed7-93eacd674a0e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:35.001330 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92377756-42e6-42c6-8ed7-93eacd674a0e admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:35.019416 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1166/2337] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 00:54:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:54:35.020030 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbf28b35-fc43-4a7a-b76b-f4ec1678f49f None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:35.030178 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbf28b35-fc43-4a7a-b76b-f4ec1678f49f None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:35.036592 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-453cd152-20dc-4bd6-888c-3fe4d71c5f64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:35.037305 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453cd152-20dc-4bd6-888c-3fe4d71c5f64 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:35.037654 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453cd152-20dc-4bd6-888c-3fe4d71c5f64 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:35.038017 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453cd152-20dc-4bd6-888c-3fe4d71c5f64 admin admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:35.038781 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-453cd152-20dc-4bd6-888c-3fe4d71c5f64 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:35.041573 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-453cd152-20dc-4bd6-888c-3fe4d71c5f64 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:35.074547 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1172/2338] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:54:35 2022] POST /identity/v3/projects => generated 389 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:54:35.086191 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92377756-42e6-42c6-8ed7-93eacd674a0e admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:35.097683 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92377756-42e6-42c6-8ed7-93eacd674a0e admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:35.173079 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e367b6-5939-4553-bcfa-3f30cb5dd287 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:35.173613 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e367b6-5939-4553-bcfa-3f30cb5dd287 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:35.173894 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e367b6-5939-4553-bcfa-3f30cb5dd287 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:35.175200 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e367b6-5939-4553-bcfa-3f30cb5dd287 admin admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:35.175200 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38e367b6-5939-4553-bcfa-3f30cb5dd287 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:35.176784 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38e367b6-5939-4553-bcfa-3f30cb5dd287 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:35.202127 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1167/2339] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:54:35 2022] POST /identity/v3/users => generated 295 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:54:35.206341 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-453cd152-20dc-4bd6-888c-3fe4d71c5f64 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:35.217741 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-453cd152-20dc-4bd6-888c-3fe4d71c5f64 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:35.222547 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07555042-cd95-415b-aa0e-44109174f9f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:35.223188 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07555042-cd95-415b-aa0e-44109174f9f6 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:35.223467 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07555042-cd95-415b-aa0e-44109174f9f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:35.223734 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07555042-cd95-415b-aa0e-44109174f9f6 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:35.224375 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07555042-cd95-415b-aa0e-44109174f9f6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:35.225969 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07555042-cd95-415b-aa0e-44109174f9f6 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:35.240936 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e367b6-5939-4553-bcfa-3f30cb5dd287 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:35.241366 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1173/2340] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:54:35 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:54:35.249460 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:35.254152 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e367b6-5939-4553-bcfa-3f30cb5dd287 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:35.261466 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73786af4-0c40-4946-b3a7-eed2b32546b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:35.262489 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73786af4-0c40-4946-b3a7-eed2b32546b1 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:35.263063 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73786af4-0c40-4946-b3a7-eed2b32546b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:35.263820 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73786af4-0c40-4946-b3a7-eed2b32546b1 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:35.264552 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73786af4-0c40-4946-b3a7-eed2b32546b1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:35.266247 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73786af4-0c40-4946-b3a7-eed2b32546b1 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:35.281424 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1168/2341] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:54:35 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:54:35.286072 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07555042-cd95-415b-aa0e-44109174f9f6 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:35.301412 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07555042-cd95-415b-aa0e-44109174f9f6 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:35.308350 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10704bea-644b-4be2-befc-234e04e0d43c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:35.310771 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10704bea-644b-4be2-befc-234e04e0d43c admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:35.311226 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10704bea-644b-4be2-befc-234e04e0d43c admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:35.311608 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10704bea-644b-4be2-befc-234e04e0d43c admin admin] PATH_INFO: `/v3/projects/c301fee2a6744802a7a3b9fcf86daf7f/users/101b25b93ce34ab58b67b1797cfdf69a/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:35.332756 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10704bea-644b-4be2-befc-234e04e0d43c admin admin] RBAC: Authorizing `identity:create_grant(project_id=c301fee2a6744802a7a3b9fcf86daf7f, user_id=101b25b93ce34ab58b67b1797cfdf69a, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:35.335527 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10704bea-644b-4be2-befc-234e04e0d43c admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:35.352615 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1174/2342] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:54:35 2022] PUT /identity/v3/projects/c301fee2a6744802a7a3b9fcf86daf7f/users/101b25b93ce34ab58b67b1797cfdf69a/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:54:35.365152 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185770d6-3e0a-4ad8-b0bd-50f017f812b6 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:35.365695 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185770d6-3e0a-4ad8-b0bd-50f017f812b6 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:35.365695 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185770d6-3e0a-4ad8-b0bd-50f017f812b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:35.367066 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7037f443-7056-4c0c-a8ea-d1a4222b9f1b None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:35.367456 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7037f443-7056-4c0c-a8ea-d1a4222b9f1b None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:35.367849 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7037f443-7056-4c0c-a8ea-d1a4222b9f1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:35.389722 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-7037f443-7056-4c0c-a8ea-d1a4222b9f1b None None] MFA Rules not processed for user `5df6979effd84ece8d75b631ba7da196`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:35.391239 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-7037f443-7056-4c0c-a8ea-d1a4222b9f1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:35.396832 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-185770d6-3e0a-4ad8-b0bd-50f017f812b6 None None] MFA Rules not processed for user `101b25b93ce34ab58b67b1797cfdf69a`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:35.398593 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-185770d6-3e0a-4ad8-b0bd-50f017f812b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:35.456869 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1175/2343] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:35 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:54:35.470469 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1169/2344] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:35 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:54:35.483407 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7037f443-7056-4c0c-a8ea-d1a4222b9f1b None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:35.484960 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-7037f443-7056-4c0c-a8ea-d1a4222b9f1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:35.553512 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7037f443-7056-4c0c-a8ea-d1a4222b9f1b None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:35.558139 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-860faffa-aa8d-4182-9f5e-a1511f614a01 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:35.558899 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860faffa-aa8d-4182-9f5e-a1511f614a01 service cinder] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:35.559276 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860faffa-aa8d-4182-9f5e-a1511f614a01 service cinder] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:35.559645 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860faffa-aa8d-4182-9f5e-a1511f614a01 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:35.570723 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-860faffa-aa8d-4182-9f5e-a1511f614a01 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:35.573222 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-860faffa-aa8d-4182-9f5e-a1511f614a01 service cinder] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:35.583990 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1176/2345] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:54:35 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:54:39.683872 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf239324-3a5c-40c9-85d3-a85d525ca798 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:39.684459 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf239324-3a5c-40c9-85d3-a85d525ca798 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:39.685620 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf239324-3a5c-40c9-85d3-a85d525ca798 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:39.744819 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-bf239324-3a5c-40c9-85d3-a85d525ca798 None None] MFA Rules not processed for user `967e0ef4e6364804a58d0d59c22cd6a8`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:39.746121 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-bf239324-3a5c-40c9-85d3-a85d525ca798 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:39.843001 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1170/2346] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:39 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:54:39.846842 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-860faffa-aa8d-4182-9f5e-a1511f614a01 service cinder] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:39.846842 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-860faffa-aa8d-4182-9f5e-a1511f614a01 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:39.896130 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=96,keystone:DELETE=2,keystone:INSERT=2 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:39.932249 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-860faffa-aa8d-4182-9f5e-a1511f614a01 service cinder] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:39.936925 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2ab0097-6039-4ca0-bea3-7ec123c5f161 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:39.937537 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ab0097-6039-4ca0-bea3-7ec123c5f161 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:39.937817 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ab0097-6039-4ca0-bea3-7ec123c5f161 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:39.938088 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ab0097-6039-4ca0-bea3-7ec123c5f161 admin admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:39.938719 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2ab0097-6039-4ca0-bea3-7ec123c5f161 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:39.941120 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2ab0097-6039-4ca0-bea3-7ec123c5f161 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:39.974831 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1177/2347] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:54:39 2022] POST /identity/v3/projects => generated 371 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:54:39.978084 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf239324-3a5c-40c9-85d3-a85d525ca798 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:39.996934 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf239324-3a5c-40c9-85d3-a85d525ca798 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:40.111857 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce1d27a-15bf-4f48-a26c-30c34f94f484 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:40.112587 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce1d27a-15bf-4f48-a26c-30c34f94f484 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:40.112842 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce1d27a-15bf-4f48-a26c-30c34f94f484 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:40.113137 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce1d27a-15bf-4f48-a26c-30c34f94f484 admin admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:40.113785 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce1d27a-15bf-4f48-a26c-30c34f94f484 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:40.115760 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce1d27a-15bf-4f48-a26c-30c34f94f484 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:40.141721 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1171/2348] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:54:39 2022] POST /identity/v3/users => generated 292 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:54:40.145672 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2ab0097-6039-4ca0-bea3-7ec123c5f161 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:40.158160 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2ab0097-6039-4ca0-bea3-7ec123c5f161 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:40.163758 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b8955c-ae8c-4419-8f62-1b5ac05eac98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:40.164374 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b8955c-ae8c-4419-8f62-1b5ac05eac98 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:40.164652 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b8955c-ae8c-4419-8f62-1b5ac05eac98 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:40.164923 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b8955c-ae8c-4419-8f62-1b5ac05eac98 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:40.166108 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0b8955c-ae8c-4419-8f62-1b5ac05eac98 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:40.167916 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0b8955c-ae8c-4419-8f62-1b5ac05eac98 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:40.180169 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1178/2349] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:54:40 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:54:40.185264 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce1d27a-15bf-4f48-a26c-30c34f94f484 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:40.190724 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:40.197635 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce1d27a-15bf-4f48-a26c-30c34f94f484 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:40.203873 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74adebd7-9419-4bd3-a655-c1ce7b261e0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:40.204740 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74adebd7-9419-4bd3-a655-c1ce7b261e0f admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:40.206317 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74adebd7-9419-4bd3-a655-c1ce7b261e0f admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:40.206673 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74adebd7-9419-4bd3-a655-c1ce7b261e0f admin admin] PATH_INFO: `/v3/projects/c0a6200260914ce48590b57b10f75f24/users/96a1830ec63b4158a2ad31bdc84f8f3c/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:40.237113 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74adebd7-9419-4bd3-a655-c1ce7b261e0f admin admin] RBAC: Authorizing `identity:create_grant(project_id=c0a6200260914ce48590b57b10f75f24, user_id=96a1830ec63b4158a2ad31bdc84f8f3c, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:40.247608 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74adebd7-9419-4bd3-a655-c1ce7b261e0f admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:40.270592 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1172/2350] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:54:40 2022] PUT /identity/v3/projects/c0a6200260914ce48590b57b10f75f24/users/96a1830ec63b4158a2ad31bdc84f8f3c/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 86 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:54:40.274886 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b8955c-ae8c-4419-8f62-1b5ac05eac98 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:40.288223 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b8955c-ae8c-4419-8f62-1b5ac05eac98 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:40.355210 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-739530b6-5aa7-4233-91c3-8bb05daadbc9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:40.355862 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739530b6-5aa7-4233-91c3-8bb05daadbc9 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:40.356142 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739530b6-5aa7-4233-91c3-8bb05daadbc9 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:40.356469 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739530b6-5aa7-4233-91c3-8bb05daadbc9 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:40.357072 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-739530b6-5aa7-4233-91c3-8bb05daadbc9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:40.358724 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-739530b6-5aa7-4233-91c3-8bb05daadbc9 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:40.370206 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1179/2351] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:54:40 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:54:40.374009 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74adebd7-9419-4bd3-a655-c1ce7b261e0f admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:40.388745 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74adebd7-9419-4bd3-a655-c1ce7b261e0f admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:40.393792 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7d21a3d-8311-40cb-aeb5-f8076495cfa0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:40.394457 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d21a3d-8311-40cb-aeb5-f8076495cfa0 admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:40.394857 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d21a3d-8311-40cb-aeb5-f8076495cfa0 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:40.395160 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d21a3d-8311-40cb-aeb5-f8076495cfa0 admin admin] PATH_INFO: `/v3/projects/c0a6200260914ce48590b57b10f75f24/users/96a1830ec63b4158a2ad31bdc84f8f3c/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:40.397075 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7d21a3d-8311-40cb-aeb5-f8076495cfa0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c0a6200260914ce48590b57b10f75f24, user_id=96a1830ec63b4158a2ad31bdc84f8f3c, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:40.399763 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7d21a3d-8311-40cb-aeb5-f8076495cfa0 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:40.411423 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1173/2352] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:54:40 2022] PUT /identity/v3/projects/c0a6200260914ce48590b57b10f75f24/users/96a1830ec63b4158a2ad31bdc84f8f3c/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:54:45.714649 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-739530b6-5aa7-4233-91c3-8bb05daadbc9 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:45.717478 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-739530b6-5aa7-4233-91c3-8bb05daadbc9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:45.751661 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-739530b6-5aa7-4233-91c3-8bb05daadbc9 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:45.872274 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de8a0ec-9755-4072-8345-670d104e64ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:45.873196 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de8a0ec-9755-4072-8345-670d104e64ec admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:45.873569 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de8a0ec-9755-4072-8345-670d104e64ec admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:45.874106 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de8a0ec-9755-4072-8345-670d104e64ec admin admin] PATH_INFO: `/v3/users/b95b1812f54d49a5a552b6b569d326ee` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:45.875586 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8de8a0ec-9755-4072-8345-670d104e64ec admin admin] RBAC: Authorizing `identity:delete_user(user_id=b95b1812f54d49a5a552b6b569d326ee)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:45.877767 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8de8a0ec-9755-4072-8345-670d104e64ec admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:45.949512 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=93,keystone:INSERT=1,keystone:DELETE=6 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:45.953230 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-8de8a0ec-9755-4072-8345-670d104e64ec admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b95b1812f54d49a5a552b6b569d326ee'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:45.955375 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-8de8a0ec-9755-4072-8345-670d104e64ec admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b95b1812f54d49a5a552b6b569d326ee'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:45.956735 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-8de8a0ec-9755-4072-8345-670d104e64ec admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b95b1812f54d49a5a552b6b569d326ee'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:45.970044 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-8de8a0ec-9755-4072-8345-670d104e64ec admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b95b1812f54d49a5a552b6b569d326ee'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:46.020779 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-8de8a0ec-9755-4072-8345-670d104e64ec admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b95b1812f54d49a5a552b6b569d326ee'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:46.069982 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1180/2353] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:54:45 2022] DELETE /identity/v3/users/b95b1812f54d49a5a552b6b569d326ee => generated 0 bytes in 356 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:54:46.075090 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2dcb95-3252-4502-8700-26ab78f45a30 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:46.075612 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2dcb95-3252-4502-8700-26ab78f45a30 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:46.075799 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2dcb95-3252-4502-8700-26ab78f45a30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:46.110764 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-1b2dcb95-3252-4502-8700-26ab78f45a30 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:46.129189 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-1b2dcb95-3252-4502-8700-26ab78f45a30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:46.253916 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1174/2354] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:54:46.289983 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de8a0ec-9755-4072-8345-670d104e64ec admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:46.292947 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-8de8a0ec-9755-4072-8345-670d104e64ec admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:46.396715 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de8a0ec-9755-4072-8345-670d104e64ec admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:46.406040 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22dcd661-6fa5-4880-a70b-cd7d465f23b7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:46.407852 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22dcd661-6fa5-4880-a70b-cd7d465f23b7 service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:46.407963 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22dcd661-6fa5-4880-a70b-cd7d465f23b7 service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:46.408272 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22dcd661-6fa5-4880-a70b-cd7d465f23b7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:46.422777 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22dcd661-6fa5-4880-a70b-cd7d465f23b7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:46.425569 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22dcd661-6fa5-4880-a70b-cd7d465f23b7 service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:46.437227 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1181/2355] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:54:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:54:46.944179 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973f609f-9a8c-4f97-b5b8-01004ad4bb69 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:46.944463 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973f609f-9a8c-4f97-b5b8-01004ad4bb69 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:46.944733 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973f609f-9a8c-4f97-b5b8-01004ad4bb69 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:46.980964 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-973f609f-9a8c-4f97-b5b8-01004ad4bb69 None None] MFA Rules not processed for user `96a1830ec63b4158a2ad31bdc84f8f3c`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:46.984046 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-973f609f-9a8c-4f97-b5b8-01004ad4bb69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:46.989288 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22dcd661-6fa5-4880-a70b-cd7d465f23b7 service neutron] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:46.990984 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-22dcd661-6fa5-4880-a70b-cd7d465f23b7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:47.006576 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22dcd661-6fa5-4880-a70b-cd7d465f23b7 service neutron] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:47.016895 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d27f1c2-eff4-4930-8c97-f224e2ccd309 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:47.018108 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d27f1c2-eff4-4930-8c97-f224e2ccd309 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:47.018643 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d27f1c2-eff4-4930-8c97-f224e2ccd309 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:47.020035 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d27f1c2-eff4-4930-8c97-f224e2ccd309 admin admin] PATH_INFO: `/v3/projects/59b634c104e744988203c197d7229e9a` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:47.023490 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d27f1c2-eff4-4930-8c97-f224e2ccd309 admin admin] RBAC: Authorizing `identity:delete_project(project_id=59b634c104e744988203c197d7229e9a)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:47.026377 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d27f1c2-eff4-4930-8c97-f224e2ccd309 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:47.073620 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1175/2356] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:46 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:54:47.089947 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973f609f-9a8c-4f97-b5b8-01004ad4bb69 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:47.092053 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-973f609f-9a8c-4f97-b5b8-01004ad4bb69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:47.107803 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-7d27f1c2-eff4-4930-8c97-f224e2ccd309 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '59b634c104e744988203c197d7229e9a', 'request_id': 'req-7d27f1c2-eff4-4930-8c97-f224e2ccd309'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:47.116007 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-7d27f1c2-eff4-4930-8c97-f224e2ccd309 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '59b634c104e744988203c197d7229e9a', 'request_id': 'req-7d27f1c2-eff4-4930-8c97-f224e2ccd309'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:47.127163 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-7d27f1c2-eff4-4930-8c97-f224e2ccd309 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '59b634c104e744988203c197d7229e9a', 'request_id': 'req-7d27f1c2-eff4-4930-8c97-f224e2ccd309'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:47.139320 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-7d27f1c2-eff4-4930-8c97-f224e2ccd309 admin admin] The token cache is being invalidate because project 59b634c104e744988203c197d7229e9a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87924) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:54:47.139521 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-7d27f1c2-eff4-4930-8c97-f224e2ccd309 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:47.140554 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-7d27f1c2-eff4-4930-8c97-f224e2ccd309 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:47.142178 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1182/2357] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:54:46 2022] DELETE /identity/v3/projects/59b634c104e744988203c197d7229e9a => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:54:47.147395 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e48a55-d232-4685-b853-a8a19982b9df None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:47.147680 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e48a55-d232-4685-b853-a8a19982b9df None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:47.147948 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e48a55-d232-4685-b853-a8a19982b9df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:47.163196 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:47.172226 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-63e48a55-d232-4685-b853-a8a19982b9df None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:47.173638 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-63e48a55-d232-4685-b853-a8a19982b9df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:47.191383 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973f609f-9a8c-4f97-b5b8-01004ad4bb69 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:47.231008 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:DELETE=11,keystone:SELECT=87,keystone:INSERT=2 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:47.259689 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1183/2358] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:54:47.268033 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9de78b9-81c7-4cc4-8a13-f6072f6145b4 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:47.268319 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9de78b9-81c7-4cc4-8a13-f6072f6145b4 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:47.268588 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9de78b9-81c7-4cc4-8a13-f6072f6145b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:47.279345 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04bc239c-ba6c-4335-be1d-2777f53df2cd service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '835def0e6c6d4365a2767051509eb0d6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'ResellerAdmin', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:47.279968 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04bc239c-ba6c-4335-be1d-2777f53df2cd service nova] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:47.280248 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04bc239c-ba6c-4335-be1d-2777f53df2cd service nova] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:47.280521 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04bc239c-ba6c-4335-be1d-2777f53df2cd service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:47.282594 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-04bc239c-ba6c-4335-be1d-2777f53df2cd service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:47.294144 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-d9de78b9-81c7-4cc4-8a13-f6072f6145b4 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:47.295865 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-d9de78b9-81c7-4cc4-8a13-f6072f6145b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:47.304735 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1184/2359] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:54:47.309177 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9de78b9-81c7-4cc4-8a13-f6072f6145b4 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:47.319441 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9de78b9-81c7-4cc4-8a13-f6072f6145b4 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:47.325260 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b8336b-02ec-426a-8cb2-a559e6d0b3e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:47.325874 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b8336b-02ec-426a-8cb2-a559e6d0b3e3 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:47.326148 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b8336b-02ec-426a-8cb2-a559e6d0b3e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:47.326418 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b8336b-02ec-426a-8cb2-a559e6d0b3e3 admin admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:47.327116 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35b8336b-02ec-426a-8cb2-a559e6d0b3e3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:54:47.327455 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35b8336b-02ec-426a-8cb2-a559e6d0b3e3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:47.329159 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35b8336b-02ec-426a-8cb2-a559e6d0b3e3 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:47.347675 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1185/2360] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 00:54:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:54:47.348727 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b8336b-02ec-426a-8cb2-a559e6d0b3e3 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:47.351061 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04bc239c-ba6c-4335-be1d-2777f53df2cd service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:47.353677 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04bc239c-ba6c-4335-be1d-2777f53df2cd service nova] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:47.361377 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b8336b-02ec-426a-8cb2-a559e6d0b3e3 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:47.365143 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1176/2361] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:54:47 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 275 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:54:47.368285 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071857fd-df94-40aa-983f-be4657e11cfa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:47.369825 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071857fd-df94-40aa-983f-be4657e11cfa admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:47.370173 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071857fd-df94-40aa-983f-be4657e11cfa admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:47.370511 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071857fd-df94-40aa-983f-be4657e11cfa admin admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:47.371379 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-071857fd-df94-40aa-983f-be4657e11cfa admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:47.373659 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-071857fd-df94-40aa-983f-be4657e11cfa admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:47.406490 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1186/2362] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:54:47 2022] POST /identity/v3/projects => generated 383 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:54:47.407739 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04bc239c-ba6c-4335-be1d-2777f53df2cd service nova] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:47.421388 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04bc239c-ba6c-4335-be1d-2777f53df2cd service nova] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:47.483907 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae50482-4509-4a91-bb6c-1ec6cd5acc11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:47.484634 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae50482-4509-4a91-bb6c-1ec6cd5acc11 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:47.484975 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae50482-4509-4a91-bb6c-1ec6cd5acc11 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:47.485312 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae50482-4509-4a91-bb6c-1ec6cd5acc11 admin admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:47.486022 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ae50482-4509-4a91-bb6c-1ec6cd5acc11 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:47.488052 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ae50482-4509-4a91-bb6c-1ec6cd5acc11 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:47.510195 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1177/2363] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:54:47 2022] POST /identity/v3/users => generated 292 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:54:47.514364 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071857fd-df94-40aa-983f-be4657e11cfa admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:47.526627 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071857fd-df94-40aa-983f-be4657e11cfa admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:47.532166 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c2b2147-a46e-48fc-8971-1d778c528eec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:47.532772 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2b2147-a46e-48fc-8971-1d778c528eec admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:47.533046 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2b2147-a46e-48fc-8971-1d778c528eec admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:47.533314 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2b2147-a46e-48fc-8971-1d778c528eec admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:47.533952 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c2b2147-a46e-48fc-8971-1d778c528eec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:47.535836 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c2b2147-a46e-48fc-8971-1d778c528eec admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:47.548376 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1187/2364] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:54:47 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:54:47.552695 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae50482-4509-4a91-bb6c-1ec6cd5acc11 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:47.567055 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae50482-4509-4a91-bb6c-1ec6cd5acc11 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:47.578114 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3474d2d6-ae36-4262-971c-d663269add9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:47.578737 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3474d2d6-ae36-4262-971c-d663269add9d admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:47.579072 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3474d2d6-ae36-4262-971c-d663269add9d admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:47.579344 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3474d2d6-ae36-4262-971c-d663269add9d admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:47.580019 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3474d2d6-ae36-4262-971c-d663269add9d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:47.581609 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3474d2d6-ae36-4262-971c-d663269add9d admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:47.593536 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1178/2365] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:54:47 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 11 00:54:47.598071 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c2b2147-a46e-48fc-8971-1d778c528eec admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:47.608962 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c2b2147-a46e-48fc-8971-1d778c528eec admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:47.614021 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-189fcd92-f372-46ad-b0cf-5ffc96de72f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:47.614682 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189fcd92-f372-46ad-b0cf-5ffc96de72f7 admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:47.614996 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189fcd92-f372-46ad-b0cf-5ffc96de72f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:47.615281 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189fcd92-f372-46ad-b0cf-5ffc96de72f7 admin admin] PATH_INFO: `/v3/projects/299fcf2ee8b24a4fbb27ed83954c8e24/users/953d7ed2081c48c79a1a5660370b861b/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:47.631106 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-189fcd92-f372-46ad-b0cf-5ffc96de72f7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=299fcf2ee8b24a4fbb27ed83954c8e24, user_id=953d7ed2081c48c79a1a5660370b861b, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:47.633843 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-189fcd92-f372-46ad-b0cf-5ffc96de72f7 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:47.648881 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1188/2366] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:54:47 2022] PUT /identity/v3/projects/299fcf2ee8b24a4fbb27ed83954c8e24/users/953d7ed2081c48c79a1a5660370b861b/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:54:47.654584 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a48f1c-e694-4bb6-ac35-17d8ee02c6a4 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:47.654918 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a48f1c-e694-4bb6-ac35-17d8ee02c6a4 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:47.655191 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a48f1c-e694-4bb6-ac35-17d8ee02c6a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:47.669252 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:47.686461 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-a7a48f1c-e694-4bb6-ac35-17d8ee02c6a4 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:47.687722 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-a7a48f1c-e694-4bb6-ac35-17d8ee02c6a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:47.769417 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1179/2367] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:54:47.783560 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-189fcd92-f372-46ad-b0cf-5ffc96de72f7 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:47.785090 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-189fcd92-f372-46ad-b0cf-5ffc96de72f7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:47.880544 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-189fcd92-f372-46ad-b0cf-5ffc96de72f7 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:47.886361 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66eeeeac-0bbc-4a4f-abb6-5094541198de service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:47.887042 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66eeeeac-0bbc-4a4f-abb6-5094541198de service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:47.887317 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66eeeeac-0bbc-4a4f-abb6-5094541198de service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:47.887590 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66eeeeac-0bbc-4a4f-abb6-5094541198de service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:47.903245 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66eeeeac-0bbc-4a4f-abb6-5094541198de service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:47.905001 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66eeeeac-0bbc-4a4f-abb6-5094541198de service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:47.918420 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1189/2368] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:54:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 11 00:54:48.758102 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5248916b-bef6-4de2-9929-67b946c28200 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:48.758595 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5248916b-bef6-4de2-9929-67b946c28200 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:48.758813 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5248916b-bef6-4de2-9929-67b946c28200 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:48.782216 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-5248916b-bef6-4de2-9929-67b946c28200 None None] MFA Rules not processed for user `967e0ef4e6364804a58d0d59c22cd6a8`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:48.783469 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-5248916b-bef6-4de2-9929-67b946c28200 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:48.853489 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1180/2369] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:48 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:54:54.408709 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f738c6e8-2ce2-4b05-bac1-21f6b8bb8f83 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:54.409313 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f738c6e8-2ce2-4b05-bac1-21f6b8bb8f83 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:54.409313 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f738c6e8-2ce2-4b05-bac1-21f6b8bb8f83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:54.449148 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-f738c6e8-2ce2-4b05-bac1-21f6b8bb8f83 None None] MFA Rules not processed for user `953d7ed2081c48c79a1a5660370b861b`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:54.450401 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-f738c6e8-2ce2-4b05-bac1-21f6b8bb8f83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:54.573533 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1190/2370] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:54 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:54:54.577973 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5248916b-bef6-4de2-9929-67b946c28200 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:54.595781 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5248916b-bef6-4de2-9929-67b946c28200 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:54.611894 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b2ff1d-fdf6-471c-bbb1-adc24e067d19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:54.613183 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b2ff1d-fdf6-471c-bbb1-adc24e067d19 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:54.613465 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b2ff1d-fdf6-471c-bbb1-adc24e067d19 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:54.613921 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b2ff1d-fdf6-471c-bbb1-adc24e067d19 admin admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:54.614731 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32b2ff1d-fdf6-471c-bbb1-adc24e067d19 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:54.616894 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32b2ff1d-fdf6-471c-bbb1-adc24e067d19 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:54.675859 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f738c6e8-2ce2-4b05-bac1-21f6b8bb8f83 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:54.685038 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1181/2371] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:54:54 2022] POST /identity/v3/projects => generated 383 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:54:54.685609 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:54.696146 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f738c6e8-2ce2-4b05-bac1-21f6b8bb8f83 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:54.813808 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5334bbc9-7849-416a-af9c-383593992fc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:54.814317 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5334bbc9-7849-416a-af9c-383593992fc3 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:54.815267 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5334bbc9-7849-416a-af9c-383593992fc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:54.815555 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5334bbc9-7849-416a-af9c-383593992fc3 admin admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:54.816230 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5334bbc9-7849-416a-af9c-383593992fc3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:54.818388 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5334bbc9-7849-416a-af9c-383593992fc3 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:54.850140 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1191/2372] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:54:54 2022] POST /identity/v3/users => generated 298 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:54:54.851099 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b2ff1d-fdf6-471c-bbb1-adc24e067d19 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:54.871635 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b2ff1d-fdf6-471c-bbb1-adc24e067d19 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:54.877391 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f46c9c81-c396-4919-bf3f-fc3950173411 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:54.878229 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f46c9c81-c396-4919-bf3f-fc3950173411 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:54.878705 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f46c9c81-c396-4919-bf3f-fc3950173411 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:54.880860 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f46c9c81-c396-4919-bf3f-fc3950173411 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:54.881753 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f46c9c81-c396-4919-bf3f-fc3950173411 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:54.883649 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f46c9c81-c396-4919-bf3f-fc3950173411 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:54.906579 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1182/2373] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:54:54 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:54:54.910489 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5334bbc9-7849-416a-af9c-383593992fc3 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:54.929860 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5334bbc9-7849-416a-af9c-383593992fc3 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:54.942121 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7d6835b-4c2d-46f3-bfef-f454884e4d57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:54.943011 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d6835b-4c2d-46f3-bfef-f454884e4d57 admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:54.943391 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d6835b-4c2d-46f3-bfef-f454884e4d57 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:54.943763 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d6835b-4c2d-46f3-bfef-f454884e4d57 admin admin] PATH_INFO: `/v3/projects/32390cc59f8540a7aac26e20f1b25166/users/6e020c4fd9534e39bb2153ecdbf5b763/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:54.975656 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f46c9c81-c396-4919-bf3f-fc3950173411 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:54.977908 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-f46c9c81-c396-4919-bf3f-fc3950173411 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:54.987039 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7d6835b-4c2d-46f3-bfef-f454884e4d57 admin admin] RBAC: Authorizing `identity:create_grant(project_id=32390cc59f8540a7aac26e20f1b25166, user_id=6e020c4fd9534e39bb2153ecdbf5b763, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:54.989917 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7d6835b-4c2d-46f3-bfef-f454884e4d57 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:55.000608 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f46c9c81-c396-4919-bf3f-fc3950173411 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:55.011479 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1adafa41-1894-4472-a7d7-ab8de2385a2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:55.012254 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1adafa41-1894-4472-a7d7-ab8de2385a2d admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:55.012959 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1adafa41-1894-4472-a7d7-ab8de2385a2d admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:55.013505 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1adafa41-1894-4472-a7d7-ab8de2385a2d admin admin] PATH_INFO: `/v3/users/101b25b93ce34ab58b67b1797cfdf69a` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:55.019055 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1192/2374] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:54:54 2022] PUT /identity/v3/projects/32390cc59f8540a7aac26e20f1b25166/users/6e020c4fd9534e39bb2153ecdbf5b763/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:54:55.022568 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1adafa41-1894-4472-a7d7-ab8de2385a2d admin admin] RBAC: Authorizing `identity:delete_user(user_id=101b25b93ce34ab58b67b1797cfdf69a)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:55.023833 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7d6835b-4c2d-46f3-bfef-f454884e4d57 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:55.042136 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1adafa41-1894-4472-a7d7-ab8de2385a2d admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:55.137175 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-1adafa41-1894-4472-a7d7-ab8de2385a2d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '101b25b93ce34ab58b67b1797cfdf69a'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:55.138341 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-1adafa41-1894-4472-a7d7-ab8de2385a2d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '101b25b93ce34ab58b67b1797cfdf69a'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:55.139307 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-1adafa41-1894-4472-a7d7-ab8de2385a2d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '101b25b93ce34ab58b67b1797cfdf69a'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:55.156747 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-1adafa41-1894-4472-a7d7-ab8de2385a2d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '101b25b93ce34ab58b67b1797cfdf69a'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:55.173234 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=90,keystone:INSERT=1,keystone:DELETE=9 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:55.175430 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-1adafa41-1894-4472-a7d7-ab8de2385a2d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '101b25b93ce34ab58b67b1797cfdf69a'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:55.189884 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7d6835b-4c2d-46f3-bfef-f454884e4d57 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:55.201995 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-125750ad-2895-4d9c-8053-35ac959e8496 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:55.202983 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125750ad-2895-4d9c-8053-35ac959e8496 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:55.203525 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125750ad-2895-4d9c-8053-35ac959e8496 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:55.203983 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125750ad-2895-4d9c-8053-35ac959e8496 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:55.204787 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-125750ad-2895-4d9c-8053-35ac959e8496 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:55.205762 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1183/2375] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:54:54 2022] DELETE /identity/v3/users/101b25b93ce34ab58b67b1797cfdf69a => generated 0 bytes in 231 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:54:55.206881 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-125750ad-2895-4d9c-8053-35ac959e8496 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:55.210112 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a94b54-2834-4c66-a9d6-c56b15fe9b4c None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:55.210476 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a94b54-2834-4c66-a9d6-c56b15fe9b4c None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:55.210862 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a94b54-2834-4c66-a9d6-c56b15fe9b4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:55.226192 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1193/2376] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:54:55 2022] GET /identity/v3/roles => generated 1420 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:54:55.227789 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3afaaf0-ab81-416d-ad28-84f677a35a78 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:55.228650 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3afaaf0-ab81-416d-ad28-84f677a35a78 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:55.228650 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3afaaf0-ab81-416d-ad28-84f677a35a78 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:55.233956 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-03a94b54-2834-4c66-a9d6-c56b15fe9b4c None None] MFA Rules not processed for user `d886b168a8eb4424987555c073d71727`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:55.235330 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-03a94b54-2834-4c66-a9d6-c56b15fe9b4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:55.248713 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-f3afaaf0-ab81-416d-ad28-84f677a35a78 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:55.251180 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-f3afaaf0-ab81-416d-ad28-84f677a35a78 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:55.308509 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1184/2377] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:55 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:54:55.309324 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a94b54-2834-4c66-a9d6-c56b15fe9b4c None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:55.314059 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-03a94b54-2834-4c66-a9d6-c56b15fe9b4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:55.323598 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1194/2378] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:54:55.333066 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a94b54-2834-4c66-a9d6-c56b15fe9b4c None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:55.333880 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3afaaf0-ab81-416d-ad28-84f677a35a78 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:55.335818 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-f3afaaf0-ab81-416d-ad28-84f677a35a78 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:55.342459 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00cd2554-3e45-4a4b-9de4-e366f467da45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:55.343526 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00cd2554-3e45-4a4b-9de4-e366f467da45 admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:55.344985 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00cd2554-3e45-4a4b-9de4-e366f467da45 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:55.345389 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00cd2554-3e45-4a4b-9de4-e366f467da45 admin admin] PATH_INFO: `/v3/projects/32390cc59f8540a7aac26e20f1b25166/users/6e020c4fd9534e39bb2153ecdbf5b763/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:55.347432 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:55.347774 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00cd2554-3e45-4a4b-9de4-e366f467da45 admin admin] RBAC: Authorizing `identity:create_grant(project_id=32390cc59f8540a7aac26e20f1b25166, user_id=6e020c4fd9534e39bb2153ecdbf5b763, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:55.351058 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00cd2554-3e45-4a4b-9de4-e366f467da45 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:55.374696 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1185/2379] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:54:55 2022] PUT /identity/v3/projects/32390cc59f8540a7aac26e20f1b25166/users/6e020c4fd9534e39bb2153ecdbf5b763/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:54:55.476411 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3afaaf0-ab81-416d-ad28-84f677a35a78 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:55.592467 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce5d63e1-0c69-412b-92c2-aab012f5c82c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:55.593213 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5d63e1-0c69-412b-92c2-aab012f5c82c service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:55.593775 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5d63e1-0c69-412b-92c2-aab012f5c82c service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:55.594150 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5d63e1-0c69-412b-92c2-aab012f5c82c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:55.715592 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce5d63e1-0c69-412b-92c2-aab012f5c82c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:55.723076 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce5d63e1-0c69-412b-92c2-aab012f5c82c service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:55.740864 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1195/2380] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:54:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 408 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 11 00:54:56.753970 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00cd2554-3e45-4a4b-9de4-e366f467da45 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:56.755912 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-00cd2554-3e45-4a4b-9de4-e366f467da45 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:56.772724 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00cd2554-3e45-4a4b-9de4-e366f467da45 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:56.781619 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d506ed-4f03-42e7-9740-063688a02d4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:56.782483 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d506ed-4f03-42e7-9740-063688a02d4e admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:56.783146 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d506ed-4f03-42e7-9740-063688a02d4e admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:56.783576 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d506ed-4f03-42e7-9740-063688a02d4e admin admin] PATH_INFO: `/v3/projects/c301fee2a6744802a7a3b9fcf86daf7f` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:56.785516 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1d506ed-4f03-42e7-9740-063688a02d4e admin admin] RBAC: Authorizing `identity:delete_project(project_id=c301fee2a6744802a7a3b9fcf86daf7f)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:56.787738 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1d506ed-4f03-42e7-9740-063688a02d4e admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:56.894679 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-e1d506ed-4f03-42e7-9740-063688a02d4e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c301fee2a6744802a7a3b9fcf86daf7f', 'request_id': 'req-e1d506ed-4f03-42e7-9740-063688a02d4e'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:56.905220 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-e1d506ed-4f03-42e7-9740-063688a02d4e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c301fee2a6744802a7a3b9fcf86daf7f', 'request_id': 'req-e1d506ed-4f03-42e7-9740-063688a02d4e'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:56.921587 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-e1d506ed-4f03-42e7-9740-063688a02d4e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c301fee2a6744802a7a3b9fcf86daf7f', 'request_id': 'req-e1d506ed-4f03-42e7-9740-063688a02d4e'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:56.940558 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-e1d506ed-4f03-42e7-9740-063688a02d4e admin admin] The token cache is being invalidate because project c301fee2a6744802a7a3b9fcf86daf7f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87923) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:54:56.940966 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-e1d506ed-4f03-42e7-9740-063688a02d4e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:56.942219 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-e1d506ed-4f03-42e7-9740-063688a02d4e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:54:56.944507 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1186/2381] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:54:56 2022] DELETE /identity/v3/projects/c301fee2a6744802a7a3b9fcf86daf7f => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:54:56.951603 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11b3de8-620c-482d-9993-a8f991cdf653 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:56.952158 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11b3de8-620c-482d-9993-a8f991cdf653 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:56.952350 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11b3de8-620c-482d-9993-a8f991cdf653 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:56.974621 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-b11b3de8-620c-482d-9993-a8f991cdf653 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:56.976691 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-b11b3de8-620c-482d-9993-a8f991cdf653 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:57.117424 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1196/2382] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:54:57.125259 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41abbc98-e661-42b2-a9c6-01aafddb9dae None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:57.125548 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41abbc98-e661-42b2-a9c6-01aafddb9dae None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:57.125842 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41abbc98-e661-42b2-a9c6-01aafddb9dae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:57.157054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-41abbc98-e661-42b2-a9c6-01aafddb9dae None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:57.158430 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-41abbc98-e661-42b2-a9c6-01aafddb9dae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:57.174584 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1187/2383] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:54:57.178551 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b11b3de8-620c-482d-9993-a8f991cdf653 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:57.192659 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b11b3de8-620c-482d-9993-a8f991cdf653 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:57.199326 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b14585a-bc47-4b5c-92e6-eebade69052c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:57.200107 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b14585a-bc47-4b5c-92e6-eebade69052c admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:57.200479 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b14585a-bc47-4b5c-92e6-eebade69052c admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:57.200846 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b14585a-bc47-4b5c-92e6-eebade69052c admin admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:57.201602 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b14585a-bc47-4b5c-92e6-eebade69052c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:54:57.201998 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b14585a-bc47-4b5c-92e6-eebade69052c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:57.209555 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b14585a-bc47-4b5c-92e6-eebade69052c admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:57.239170 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:57.252095 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1197/2384] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 00:54:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 74 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:54:57.255941 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41abbc98-e661-42b2-a9c6-01aafddb9dae None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:57.270712 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41abbc98-e661-42b2-a9c6-01aafddb9dae None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:57.297394 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e7b8c72-80cb-4589-b9bf-ae1158630c76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:57.298265 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7b8c72-80cb-4589-b9bf-ae1158630c76 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:57.299326 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7b8c72-80cb-4589-b9bf-ae1158630c76 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:57.299326 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7b8c72-80cb-4589-b9bf-ae1158630c76 admin admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:57.299730 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e7b8c72-80cb-4589-b9bf-ae1158630c76 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:57.301747 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e7b8c72-80cb-4589-b9bf-ae1158630c76 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:57.363318 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1188/2385] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:54:57 2022] POST /identity/v3/projects => generated 375 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:54:57.367553 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b14585a-bc47-4b5c-92e6-eebade69052c admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:57.380082 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b14585a-bc47-4b5c-92e6-eebade69052c admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:57.473705 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81fa7e39-12fb-4d3e-b75f-8913d9244b29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:57.474679 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81fa7e39-12fb-4d3e-b75f-8913d9244b29 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:57.475374 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81fa7e39-12fb-4d3e-b75f-8913d9244b29 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:57.475816 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81fa7e39-12fb-4d3e-b75f-8913d9244b29 admin admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:57.476710 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81fa7e39-12fb-4d3e-b75f-8913d9244b29 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:57.479943 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81fa7e39-12fb-4d3e-b75f-8913d9244b29 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:57.517657 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1198/2386] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:54:57 2022] POST /identity/v3/users => generated 288 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:54:57.522197 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e7b8c72-80cb-4589-b9bf-ae1158630c76 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:57.539125 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e7b8c72-80cb-4589-b9bf-ae1158630c76 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:57.547685 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706d95f7-5cc8-419a-8331-699463133585 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:57.548340 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706d95f7-5cc8-419a-8331-699463133585 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:57.548599 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706d95f7-5cc8-419a-8331-699463133585 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:57.548898 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706d95f7-5cc8-419a-8331-699463133585 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:57.550321 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-706d95f7-5cc8-419a-8331-699463133585 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:57.558405 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-706d95f7-5cc8-419a-8331-699463133585 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:57.580298 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1189/2387] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:54:57 2022] GET /identity/v3/roles => generated 1420 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:54:57.583949 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81fa7e39-12fb-4d3e-b75f-8913d9244b29 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:57.611206 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81fa7e39-12fb-4d3e-b75f-8913d9244b29 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:57.617983 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf3a983-6bf1-47c6-967f-4cea0044726a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:57.620002 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf3a983-6bf1-47c6-967f-4cea0044726a admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:57.620537 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf3a983-6bf1-47c6-967f-4cea0044726a admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:57.621330 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf3a983-6bf1-47c6-967f-4cea0044726a admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:57.622141 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf3a983-6bf1-47c6-967f-4cea0044726a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:57.624316 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf3a983-6bf1-47c6-967f-4cea0044726a admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:57.642362 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1199/2388] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:54:57 2022] GET /identity/v3/roles => generated 1420 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 11 00:54:57.646574 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706d95f7-5cc8-419a-8331-699463133585 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:57.662732 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706d95f7-5cc8-419a-8331-699463133585 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:57.671142 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ee586c4-41ed-4b69-98e9-d57a6ec42e7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:57.671993 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee586c4-41ed-4b69-98e9-d57a6ec42e7d admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:57.672375 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee586c4-41ed-4b69-98e9-d57a6ec42e7d admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:57.672749 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee586c4-41ed-4b69-98e9-d57a6ec42e7d admin admin] PATH_INFO: `/v3/projects/a1368870ddbd49be857e9ac128bc684e/users/e1012f856881485cb40d916131a5d763/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:57.701675 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ee586c4-41ed-4b69-98e9-d57a6ec42e7d admin admin] RBAC: Authorizing `identity:create_grant(project_id=a1368870ddbd49be857e9ac128bc684e, user_id=e1012f856881485cb40d916131a5d763, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:57.704480 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ee586c4-41ed-4b69-98e9-d57a6ec42e7d admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:57.718337 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=87,keystone:INSERT=5,keystone:DELETE=8 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:54:57.727323 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1190/2389] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:54:57 2022] PUT /identity/v3/projects/a1368870ddbd49be857e9ac128bc684e/users/e1012f856881485cb40d916131a5d763/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 81 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:54:57.733132 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdce52f1-3e62-4508-baa1-108dc9a215db None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:57.733720 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdce52f1-3e62-4508-baa1-108dc9a215db None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:57.734171 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdce52f1-3e62-4508-baa1-108dc9a215db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:57.777119 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-bdce52f1-3e62-4508-baa1-108dc9a215db None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:54:57.778705 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-bdce52f1-3e62-4508-baa1-108dc9a215db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:57.908719 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1200/2390] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:54:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:54:57.928491 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ee586c4-41ed-4b69-98e9-d57a6ec42e7d admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:54:57.931375 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-0ee586c4-41ed-4b69-98e9-d57a6ec42e7d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:54:58.086839 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ee586c4-41ed-4b69-98e9-d57a6ec42e7d admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:54:58.097434 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe3c29b6-987f-46c0-baca-61978c91b2ac service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:54:58.102777 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3c29b6-987f-46c0-baca-61978c91b2ac service neutron] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:54:58.104348 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3c29b6-987f-46c0-baca-61978c91b2ac service neutron] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:54:58.106163 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3c29b6-987f-46c0-baca-61978c91b2ac service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:54:58.140217 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe3c29b6-987f-46c0-baca-61978c91b2ac service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:54:58.141900 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe3c29b6-987f-46c0-baca-61978c91b2ac service neutron] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:54:58.166519 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1191/2391] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:54:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 239 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:55:04.365605 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd67787-2b6d-4e6d-8f38-2f231dda3091 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:55:04.366134 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd67787-2b6d-4e6d-8f38-2f231dda3091 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:55:04.366134 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd67787-2b6d-4e6d-8f38-2f231dda3091 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:55:04.413782 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-4fd67787-2b6d-4e6d-8f38-2f231dda3091 None None] MFA Rules not processed for user `9fc4274836844ae4ad5d16d74c36f671`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:55:04.416891 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-4fd67787-2b6d-4e6d-8f38-2f231dda3091 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:55:04.561946 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1201/2392] 10.222.0.27 () {58 vars in 1009 bytes} [Sun Sep 11 00:55:04 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:55:04.618376 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe3c29b6-987f-46c0-baca-61978c91b2ac service neutron] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:55:04.620508 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-fe3c29b6-987f-46c0-baca-61978c91b2ac service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:55:04.788490 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe3c29b6-987f-46c0-baca-61978c91b2ac service neutron] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:55:04.799736 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea1d252-2ca0-4443-b18f-b1afde0eb3c5 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bcb73dec73464a3abc4b774abb4f416f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:55:04.800376 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea1d252-2ca0-4443-b18f-b1afde0eb3c5 service swift] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:55:04.800655 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea1d252-2ca0-4443-b18f-b1afde0eb3c5 service swift] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:55:04.800928 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea1d252-2ca0-4443-b18f-b1afde0eb3c5 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:55:04.826283 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dea1d252-2ca0-4443-b18f-b1afde0eb3c5 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:55:04.828462 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dea1d252-2ca0-4443-b18f-b1afde0eb3c5 service swift] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:55:04.840619 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1192/2393] 10.222.0.27 () {60 vars in 1384 bytes} [Sun Sep 11 00:55:04 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 223 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 11 00:55:06.112423 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a872f373-c8a2-4204-ace4-11fdc31c6cbb None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:55:06.113007 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a872f373-c8a2-4204-ace4-11fdc31c6cbb None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:55:06.113007 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a872f373-c8a2-4204-ace4-11fdc31c6cbb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:55:06.184564 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-a872f373-c8a2-4204-ace4-11fdc31c6cbb None None] MFA Rules not processed for user `6e020c4fd9534e39bb2153ecdbf5b763`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:55:06.185984 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-a872f373-c8a2-4204-ace4-11fdc31c6cbb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:55:06.226177 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:55:06.405939 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1202/2394] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:55:06 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 294 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:55:06.415220 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea1d252-2ca0-4443-b18f-b1afde0eb3c5 service swift] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:55:06.417264 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-dea1d252-2ca0-4443-b18f-b1afde0eb3c5 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:55:06.529359 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea1d252-2ca0-4443-b18f-b1afde0eb3c5 service swift] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:55:06.533495 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f227c5-9d32-4c33-a01f-b5d0c938722c service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '06b1c911731f4267a5f37e91d948e7e9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:55:06.534128 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f227c5-9d32-4c33-a01f-b5d0c938722c service glance] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:55:06.534650 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f227c5-9d32-4c33-a01f-b5d0c938722c service glance] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:55:06.534965 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f227c5-9d32-4c33-a01f-b5d0c938722c service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:55:06.536753 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-d9f227c5-9d32-4c33-a01f-b5d0c938722c service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:55:06.601948 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:55:06.631580 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9f227c5-9d32-4c33-a01f-b5d0c938722c service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:55:06.634048 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9f227c5-9d32-4c33-a01f-b5d0c938722c service glance] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:55:06.645134 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1193/2395] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:55:06 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 230 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:55:07.038689 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb475a9-0c8e-49cd-adc4-c496231e8dfb None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:55:07.039009 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb475a9-0c8e-49cd-adc4-c496231e8dfb None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:55:07.039284 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb475a9-0c8e-49cd-adc4-c496231e8dfb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:55:07.078728 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-1bb475a9-0c8e-49cd-adc4-c496231e8dfb None None] MFA Rules not processed for user `9fc4274836844ae4ad5d16d74c36f671`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:55:07.081608 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-1bb475a9-0c8e-49cd-adc4-c496231e8dfb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:55:07.112617 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1203/2396] 10.222.0.27 () {58 vars in 1009 bytes} [Sun Sep 11 00:55:07 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:55:07.129496 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f227c5-9d32-4c33-a01f-b5d0c938722c service glance] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:55:07.161274 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f227c5-9d32-4c33-a01f-b5d0c938722c service glance] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:55:07.195462 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b40548-9713-4d49-80fa-097869671611 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bcb73dec73464a3abc4b774abb4f416f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:55:07.202170 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b40548-9713-4d49-80fa-097869671611 service swift] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:55:07.202458 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b40548-9713-4d49-80fa-097869671611 service swift] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:55:07.206844 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b40548-9713-4d49-80fa-097869671611 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:55:07.256043 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72b40548-9713-4d49-80fa-097869671611 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:55:07.257687 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72b40548-9713-4d49-80fa-097869671611 service swift] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:55:07.309623 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1194/2397] 10.222.0.27 () {60 vars in 1384 bytes} [Sun Sep 11 00:55:07 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 181 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 11 00:55:12.224655 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9bb4b99-8633-4304-909a-0f1495367298 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:55:12.225204 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9bb4b99-8633-4304-909a-0f1495367298 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:55:12.225204 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9bb4b99-8633-4304-909a-0f1495367298 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:55:12.301869 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-f9bb4b99-8633-4304-909a-0f1495367298 None None] MFA Rules not processed for user `e1012f856881485cb40d916131a5d763`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:55:12.303854 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-f9bb4b99-8633-4304-909a-0f1495367298 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:55:12.518367 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b40548-9713-4d49-80fa-097869671611 service swift] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:55:12.520757 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1204/2398] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:55:12 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 297 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:55:12.556841 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b40548-9713-4d49-80fa-097869671611 service swift] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:55:12.580619 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee0d6f2-c35c-48c8-af44-85f444e67ca9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:55:12.583914 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee0d6f2-c35c-48c8-af44-85f444e67ca9 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:55:12.584860 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee0d6f2-c35c-48c8-af44-85f444e67ca9 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:55:12.585983 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee0d6f2-c35c-48c8-af44-85f444e67ca9 admin admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:55:12.587615 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee0d6f2-c35c-48c8-af44-85f444e67ca9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:55:12.593736 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee0d6f2-c35c-48c8-af44-85f444e67ca9 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:55:12.709207 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9bb4b99-8633-4304-909a-0f1495367298 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:55:12.720290 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1195/2399] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:55:12 2022] POST /identity/v3/projects => generated 379 bytes in 200 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:55:12.727501 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9bb4b99-8633-4304-909a-0f1495367298 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:55:12.918609 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ded0fc-47ab-431e-8448-50ad69bbdc1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:55:12.921890 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ded0fc-47ab-431e-8448-50ad69bbdc1b admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:55:12.922114 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ded0fc-47ab-431e-8448-50ad69bbdc1b admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:55:12.922386 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ded0fc-47ab-431e-8448-50ad69bbdc1b admin admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:55:12.925624 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ded0fc-47ab-431e-8448-50ad69bbdc1b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:55:12.936189 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ded0fc-47ab-431e-8448-50ad69bbdc1b admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:55:13.010227 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee0d6f2-c35c-48c8-af44-85f444e67ca9 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:55:13.024836 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1205/2400] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:55:12 2022] POST /identity/v3/users => generated 296 bytes in 315 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:55:13.035275 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee0d6f2-c35c-48c8-af44-85f444e67ca9 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:55:13.056528 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-130ddc50-5f14-4072-b265-9672eaf5a4ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:55:13.057926 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130ddc50-5f14-4072-b265-9672eaf5a4ca admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:55:13.058120 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130ddc50-5f14-4072-b265-9672eaf5a4ca admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:55:13.059927 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130ddc50-5f14-4072-b265-9672eaf5a4ca admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:55:13.061203 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-130ddc50-5f14-4072-b265-9672eaf5a4ca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:55:13.065741 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-130ddc50-5f14-4072-b265-9672eaf5a4ca admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:55:13.108340 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ded0fc-47ab-431e-8448-50ad69bbdc1b admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:55:13.108672 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1196/2401] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:55:13 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:55:13.178775 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ded0fc-47ab-431e-8448-50ad69bbdc1b admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:55:13.185755 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42daeee-567a-4298-a9d5-d53ba0d82b33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:55:13.186404 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42daeee-567a-4298-a9d5-d53ba0d82b33 admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:55:13.186683 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42daeee-567a-4298-a9d5-d53ba0d82b33 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:55:13.187110 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42daeee-567a-4298-a9d5-d53ba0d82b33 admin admin] PATH_INFO: `/v3/projects/1ef38a4efe0a41548e5ae4a9d7d414ea/users/39a914ebb07648fdb5cfd683da419e11/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:55:13.206583 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f42daeee-567a-4298-a9d5-d53ba0d82b33 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1ef38a4efe0a41548e5ae4a9d7d414ea, user_id=39a914ebb07648fdb5cfd683da419e11, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:55:13.209523 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f42daeee-567a-4298-a9d5-d53ba0d82b33 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:55:13.216003 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:55:13.245706 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1206/2402] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:55:13 2022] PUT /identity/v3/projects/1ef38a4efe0a41548e5ae4a9d7d414ea/users/39a914ebb07648fdb5cfd683da419e11/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:55:13.250176 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-130ddc50-5f14-4072-b265-9672eaf5a4ca admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:55:13.268301 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-130ddc50-5f14-4072-b265-9672eaf5a4ca admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:55:13.454871 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa7fe16-d2c3-4030-aab0-1d93f8472613 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:55:13.455420 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa7fe16-d2c3-4030-aab0-1d93f8472613 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:55:13.455727 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa7fe16-d2c3-4030-aab0-1d93f8472613 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:55:13.455951 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa7fe16-d2c3-4030-aab0-1d93f8472613 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:55:13.456595 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fa7fe16-d2c3-4030-aab0-1d93f8472613 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:55:13.464748 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fa7fe16-d2c3-4030-aab0-1d93f8472613 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:55:13.530594 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1197/2403] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:55:13 2022] GET /identity/v3/roles => generated 1420 bytes in 281 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:55:13.536492 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42daeee-567a-4298-a9d5-d53ba0d82b33 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:55:13.563239 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42daeee-567a-4298-a9d5-d53ba0d82b33 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:55:13.587974 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f209848c-2311-49f3-945d-cc5bb1b177ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:55:13.591427 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f209848c-2311-49f3-945d-cc5bb1b177ac admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:55:13.592024 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f209848c-2311-49f3-945d-cc5bb1b177ac admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:55:13.592606 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f209848c-2311-49f3-945d-cc5bb1b177ac admin admin] PATH_INFO: `/v3/projects/1ef38a4efe0a41548e5ae4a9d7d414ea/users/39a914ebb07648fdb5cfd683da419e11/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:55:13.607803 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f209848c-2311-49f3-945d-cc5bb1b177ac admin admin] RBAC: Authorizing `identity:create_grant(project_id=1ef38a4efe0a41548e5ae4a9d7d414ea, user_id=39a914ebb07648fdb5cfd683da419e11, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:55:13.624634 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f209848c-2311-49f3-945d-cc5bb1b177ac admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:55:13.692182 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1207/2404] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:55:13 2022] PUT /identity/v3/projects/1ef38a4efe0a41548e5ae4a9d7d414ea/users/39a914ebb07648fdb5cfd683da419e11/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:55:23.498349 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=55,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:55:23.659675 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=7,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:55:27.572574 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf6afed-78c5-431f-b4d9-9302e46e987a None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:55:27.573220 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf6afed-78c5-431f-b4d9-9302e46e987a None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:55:27.573220 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf6afed-78c5-431f-b4d9-9302e46e987a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:55:27.658678 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-8cf6afed-78c5-431f-b4d9-9302e46e987a None None] MFA Rules not processed for user `39a914ebb07648fdb5cfd683da419e11`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:55:27.663168 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-8cf6afed-78c5-431f-b4d9-9302e46e987a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:55:27.875092 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1198/2405] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:55:27 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 304 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:55:27.888821 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f209848c-2311-49f3-945d-cc5bb1b177ac admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:55:27.891119 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-f209848c-2311-49f3-945d-cc5bb1b177ac admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:55:28.002743 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f209848c-2311-49f3-945d-cc5bb1b177ac admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:55:28.009141 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce54c603-cde9-4af5-8343-3063263c7869 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '835def0e6c6d4365a2767051509eb0d6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'ResellerAdmin', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:55:28.010131 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce54c603-cde9-4af5-8343-3063263c7869 service nova] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:55:28.010638 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce54c603-cde9-4af5-8343-3063263c7869 service nova] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:55:28.011303 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce54c603-cde9-4af5-8343-3063263c7869 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:55:28.092377 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce54c603-cde9-4af5-8343-3063263c7869 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:55:28.095298 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce54c603-cde9-4af5-8343-3063263c7869 service nova] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:55:28.110645 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1208/2406] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:55:27 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 222 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:55:28.656159 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4b7116-2435-4da9-99b4-3573445988e8 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:55:28.656731 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4b7116-2435-4da9-99b4-3573445988e8 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:55:28.656844 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4b7116-2435-4da9-99b4-3573445988e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:55:28.677185 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-2c4b7116-2435-4da9-99b4-3573445988e8 None None] MFA Rules not processed for user `953d7ed2081c48c79a1a5660370b861b`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:55:28.678695 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-2c4b7116-2435-4da9-99b4-3573445988e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:55:28.798555 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1199/2407] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:55:28 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:55:33.886693 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7bc2265-7aee-41e3-b984-c0d502cba987 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:55:33.887266 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7bc2265-7aee-41e3-b984-c0d502cba987 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:55:33.887356 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7bc2265-7aee-41e3-b984-c0d502cba987 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:55:33.889443 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1209/2408] 10.222.0.27 () {54 vars in 921 bytes} [Sun Sep 11 00:55:33 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:55:33.897678 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a45621-726f-42ca-9a89-ce050a44eed4 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:55:33.898388 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a45621-726f-42ca-9a89-ce050a44eed4 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:55:33.898547 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a45621-726f-42ca-9a89-ce050a44eed4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:55:33.920259 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-d0a45621-726f-42ca-9a89-ce050a44eed4 None None] MFA Rules not processed for user `835def0e6c6d4365a2767051509eb0d6`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:55:33.921566 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-d0a45621-726f-42ca-9a89-ce050a44eed4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:55:33.934536 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1200/2409] 10.222.0.27 () {58 vars in 1018 bytes} [Sun Sep 11 00:55:33 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:55:33.947534 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7bc2265-7aee-41e3-b984-c0d502cba987 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:55:34.026572 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7bc2265-7aee-41e3-b984-c0d502cba987 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:55:34.030283 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16166099-faaa-4c8e-bba0-41234077cf43 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '06b1c911731f4267a5f37e91d948e7e9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:55:34.030928 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16166099-faaa-4c8e-bba0-41234077cf43 service glance] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:55:34.031210 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16166099-faaa-4c8e-bba0-41234077cf43 service glance] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:55:34.031484 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16166099-faaa-4c8e-bba0-41234077cf43 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:55:34.049179 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16166099-faaa-4c8e-bba0-41234077cf43 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:55:34.051598 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16166099-faaa-4c8e-bba0-41234077cf43 service glance] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:55:34.066845 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1210/2410] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:55:33 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:55:34.151745 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e78c66-474c-469f-9455-ba1781f48ce0 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:55:34.152290 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e78c66-474c-469f-9455-ba1781f48ce0 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:55:34.152775 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e78c66-474c-469f-9455-ba1781f48ce0 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:55:34.154363 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1201/2411] 10.222.0.27 () {54 vars in 921 bytes} [Sun Sep 11 00:55:34 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:55:34.162851 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67eab72a-cdfa-48db-8551-3f2f018ecf1d None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:55:34.163678 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67eab72a-cdfa-48db-8551-3f2f018ecf1d None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:55:34.163966 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67eab72a-cdfa-48db-8551-3f2f018ecf1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:55:34.192261 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-67eab72a-cdfa-48db-8551-3f2f018ecf1d None None] MFA Rules not processed for user `a5f7b13283cb4ebfb25160aa6ad0e2f4`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:55:34.193550 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-67eab72a-cdfa-48db-8551-3f2f018ecf1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:55:34.242949 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:55:34.301975 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1211/2412] 10.222.0.27 () {58 vars in 1018 bytes} [Sun Sep 11 00:55:34 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:55:34.318842 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e78c66-474c-469f-9455-ba1781f48ce0 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:55:34.334752 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e78c66-474c-469f-9455-ba1781f48ce0 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:55:34.339142 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-699a8a16-b8cb-419d-b998-023031db2771 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:55:34.340024 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699a8a16-b8cb-419d-b998-023031db2771 service neutron] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:55:34.340361 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699a8a16-b8cb-419d-b998-023031db2771 service neutron] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:55:34.340695 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699a8a16-b8cb-419d-b998-023031db2771 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:55:34.351756 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-699a8a16-b8cb-419d-b998-023031db2771 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:55:34.353601 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-699a8a16-b8cb-419d-b998-023031db2771 service neutron] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:55:34.362599 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1202/2413] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:55:34 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:55:34.423103 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67eab72a-cdfa-48db-8551-3f2f018ecf1d None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:55:34.424681 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-67eab72a-cdfa-48db-8551-3f2f018ecf1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:55:34.438357 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67eab72a-cdfa-48db-8551-3f2f018ecf1d None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:55:34.443387 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fee9879-769c-4548-8010-a100db6e7f43 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:55:34.444167 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fee9879-769c-4548-8010-a100db6e7f43 service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:55:34.444587 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fee9879-769c-4548-8010-a100db6e7f43 service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:55:34.445122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fee9879-769c-4548-8010-a100db6e7f43 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:55:34.447072 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-1fee9879-769c-4548-8010-a100db6e7f43 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:55:34.529223 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fee9879-769c-4548-8010-a100db6e7f43 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:55:34.531072 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fee9879-769c-4548-8010-a100db6e7f43 service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:55:34.537898 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e03fd26-3e6e-47d3-b84f-6e61066f711b None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:55:34.538940 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e03fd26-3e6e-47d3-b84f-6e61066f711b None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:55:34.539390 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e03fd26-3e6e-47d3-b84f-6e61066f711b None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:55:34.540909 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1203/2414] 10.222.0.27 () {54 vars in 921 bytes} [Sun Sep 11 00:55:34 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:55:34.548436 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1212/2415] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:55:34 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:55:34.553594 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6521f50-39c7-4592-8915-a26cf6ae9fc9 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:55:34.554878 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6521f50-39c7-4592-8915-a26cf6ae9fc9 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:55:34.555282 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6521f50-39c7-4592-8915-a26cf6ae9fc9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:55:34.586504 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-c6521f50-39c7-4592-8915-a26cf6ae9fc9 None None] MFA Rules not processed for user `835def0e6c6d4365a2767051509eb0d6`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:55:34.588048 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-c6521f50-39c7-4592-8915-a26cf6ae9fc9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:55:34.602417 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1204/2416] 10.222.0.27 () {58 vars in 1018 bytes} [Sun Sep 11 00:55:34 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:55:34.620003 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fee9879-769c-4548-8010-a100db6e7f43 service neutron] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:55:34.638163 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fee9879-769c-4548-8010-a100db6e7f43 service neutron] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:55:34.642706 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1defd1b8-e04c-4ad8-bdf0-67f11ffe0c41 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:55:34.643426 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1defd1b8-e04c-4ad8-bdf0-67f11ffe0c41 service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:55:34.643774 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1defd1b8-e04c-4ad8-bdf0-67f11ffe0c41 service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:55:34.644114 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1defd1b8-e04c-4ad8-bdf0-67f11ffe0c41 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:55:34.656722 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1defd1b8-e04c-4ad8-bdf0-67f11ffe0c41 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:55:34.658474 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1defd1b8-e04c-4ad8-bdf0-67f11ffe0c41 service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:55:34.675829 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1213/2417] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:55:34 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 58 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:55:35.166243 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2411924e-deb6-4b3e-bc47-ff17c9b65dba None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:55:35.167267 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2411924e-deb6-4b3e-bc47-ff17c9b65dba None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:55:35.167541 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2411924e-deb6-4b3e-bc47-ff17c9b65dba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:55:35.192308 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-2411924e-deb6-4b3e-bc47-ff17c9b65dba None None] MFA Rules not processed for user `9fc4274836844ae4ad5d16d74c36f671`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:55:35.193799 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-2411924e-deb6-4b3e-bc47-ff17c9b65dba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:55:35.288583 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1205/2418] 10.222.0.27 () {58 vars in 1009 bytes} [Sun Sep 11 00:55:35 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:55:35.304705 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1defd1b8-e04c-4ad8-bdf0-67f11ffe0c41 service neutron] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:55:35.429673 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1defd1b8-e04c-4ad8-bdf0-67f11ffe0c41 service neutron] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:55:35.435868 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d32b9c79-5879-41cf-9f51-1fa45fb47000 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bcb73dec73464a3abc4b774abb4f416f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:55:35.436814 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32b9c79-5879-41cf-9f51-1fa45fb47000 service swift] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:55:35.437156 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32b9c79-5879-41cf-9f51-1fa45fb47000 service swift] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:55:35.437512 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32b9c79-5879-41cf-9f51-1fa45fb47000 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:55:35.447984 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d32b9c79-5879-41cf-9f51-1fa45fb47000 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:55:35.449445 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d32b9c79-5879-41cf-9f51-1fa45fb47000 service swift] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:55:35.462456 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1214/2419] 10.222.0.27 () {60 vars in 1384 bytes} [Sun Sep 11 00:55:35 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 159 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 11 00:55:45.279977 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=96 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:55:45.456888 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=73 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:55:48.308777 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8d1e4a-de3f-4710-bf0f-3d6e086ed016 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:55:48.309367 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8d1e4a-de3f-4710-bf0f-3d6e086ed016 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:55:48.309367 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8d1e4a-de3f-4710-bf0f-3d6e086ed016 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:55:48.328727 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-8b8d1e4a-de3f-4710-bf0f-3d6e086ed016 None None] MFA Rules not processed for user `e1012f856881485cb40d916131a5d763`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:55:48.329920 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-8b8d1e4a-de3f-4710-bf0f-3d6e086ed016 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:55:48.337516 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1206/2420] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:55:48 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:55:49.921433 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d32b9c79-5879-41cf-9f51-1fa45fb47000 service swift] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:55:49.922853 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-d32b9c79-5879-41cf-9f51-1fa45fb47000 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:55:50.000939 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538696ff-193e-4eb8-94d7-2358674a30fa None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:55:50.001471 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538696ff-193e-4eb8-94d7-2358674a30fa None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:55:50.001733 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538696ff-193e-4eb8-94d7-2358674a30fa None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:55:50.003578 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1207/2421] 10.222.0.27 () {54 vars in 961 bytes} [Sun Sep 11 00:55:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 11 00:55:50.010148 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51bd4f3-f558-4cea-8952-20463078dfb9 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:55:50.010519 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51bd4f3-f558-4cea-8952-20463078dfb9 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:55:50.010944 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51bd4f3-f558-4cea-8952-20463078dfb9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:55:50.014273 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d32b9c79-5879-41cf-9f51-1fa45fb47000 service swift] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:55:50.023961 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e774d11-ebb0-4c0a-bb87-e2d7c13979c3 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cdbfc2ccd73e4a5eb8c999be5b094619', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:55:50.024779 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e774d11-ebb0-4c0a-bb87-e2d7c13979c3 service placement] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:55:50.025228 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e774d11-ebb0-4c0a-bb87-e2d7c13979c3 service placement] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:55:50.025827 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e774d11-ebb0-4c0a-bb87-e2d7c13979c3 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:55:50.028172 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-0e774d11-ebb0-4c0a-bb87-e2d7c13979c3 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:55:50.045036 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e774d11-ebb0-4c0a-bb87-e2d7c13979c3 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:55:50.046943 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e774d11-ebb0-4c0a-bb87-e2d7c13979c3 service placement] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:55:50.057587 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-d51bd4f3-f558-4cea-8952-20463078dfb9 None None] MFA Rules not processed for user `cdbfc2ccd73e4a5eb8c999be5b094619`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:55:50.061732 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-d51bd4f3-f558-4cea-8952-20463078dfb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:55:50.075704 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1215/2422] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:55:49 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 154 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:55:50.096937 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1208/2423] 10.222.0.27 () {58 vars in 1058 bytes} [Sun Sep 11 00:55:50 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:55:50.098820 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e774d11-ebb0-4c0a-bb87-e2d7c13979c3 service placement] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:55:50.118758 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e774d11-ebb0-4c0a-bb87-e2d7c13979c3 service placement] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:55:50.130472 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a1482ff-793f-44e6-8802-911d60ae26c1 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cdbfc2ccd73e4a5eb8c999be5b094619', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:55:50.131154 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a1482ff-793f-44e6-8802-911d60ae26c1 service placement] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:55:50.131549 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a1482ff-793f-44e6-8802-911d60ae26c1 service placement] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:55:50.131844 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a1482ff-793f-44e6-8802-911d60ae26c1 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:55:50.141574 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a1482ff-793f-44e6-8802-911d60ae26c1 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:55:50.143228 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a1482ff-793f-44e6-8802-911d60ae26c1 service placement] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:55:50.153710 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1216/2424] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:55:50 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 57 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 11 00:55:55.183070 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f12f39-5501-4261-a175-3a4f66666383 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:55:55.183570 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f12f39-5501-4261-a175-3a4f66666383 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:55:55.183684 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f12f39-5501-4261-a175-3a4f66666383 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:55:55.203784 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-d7f12f39-5501-4261-a175-3a4f66666383 None None] MFA Rules not processed for user `9fc4274836844ae4ad5d16d74c36f671`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:55:55.204975 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-d7f12f39-5501-4261-a175-3a4f66666383 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:55:55.213155 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1209/2425] 10.222.0.27 () {58 vars in 1009 bytes} [Sun Sep 11 00:55:55 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:55:55.227790 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a1482ff-793f-44e6-8802-911d60ae26c1 service placement] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:55:55.250685 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a1482ff-793f-44e6-8802-911d60ae26c1 service placement] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:55:55.268278 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a833da8-ee8c-4a82-b41c-52f0761f1522 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bcb73dec73464a3abc4b774abb4f416f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:55:55.268905 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a833da8-ee8c-4a82-b41c-52f0761f1522 service swift] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:55:55.269210 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a833da8-ee8c-4a82-b41c-52f0761f1522 service swift] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:55:55.269482 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a833da8-ee8c-4a82-b41c-52f0761f1522 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:55:55.286249 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a833da8-ee8c-4a82-b41c-52f0761f1522 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:55:55.287827 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a833da8-ee8c-4a82-b41c-52f0761f1522 service swift] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:55:55.297625 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1217/2426] 10.222.0.27 () {60 vars in 1384 bytes} [Sun Sep 11 00:55:55 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 70 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 11 00:56:05.198089 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=9 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:56:05.293882 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=44 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:56:11.408510 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e2fc53-89ad-4fcf-bcca-52b179a06b27 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:56:11.409318 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e2fc53-89ad-4fcf-bcca-52b179a06b27 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:56:11.409318 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e2fc53-89ad-4fcf-bcca-52b179a06b27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:56:11.430105 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-90e2fc53-89ad-4fcf-bcca-52b179a06b27 None None] MFA Rules not processed for user `5df6979effd84ece8d75b631ba7da196`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:56:11.431334 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-90e2fc53-89ad-4fcf-bcca-52b179a06b27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:56:11.492361 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1210/2427] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:56:11 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:56:21.485558 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=25 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:56:22.047734 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0cfb15a-fa26-4fa8-97f4-41ed2ee02d14 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:56:22.048322 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0cfb15a-fa26-4fa8-97f4-41ed2ee02d14 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:56:22.048322 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0cfb15a-fa26-4fa8-97f4-41ed2ee02d14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:56:22.096716 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-b0cfb15a-fa26-4fa8-97f4-41ed2ee02d14 None None] MFA Rules not processed for user `9fc4274836844ae4ad5d16d74c36f671`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:56:22.097947 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-b0cfb15a-fa26-4fa8-97f4-41ed2ee02d14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:56:22.112062 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1218/2428] 10.222.0.27 () {58 vars in 967 bytes} [Sun Sep 11 00:56:22 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 65 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:56:22.146248 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e2fc53-89ad-4fcf-bcca-52b179a06b27 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:56:22.163252 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e2fc53-89ad-4fcf-bcca-52b179a06b27 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:56:22.169468 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7fb14b-5ab6-4fe5-8465-3aeda732105d service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bcb73dec73464a3abc4b774abb4f416f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:56:22.170082 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7fb14b-5ab6-4fe5-8465-3aeda732105d service swift] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:56:22.170511 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7fb14b-5ab6-4fe5-8465-3aeda732105d service swift] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:56:22.170813 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7fb14b-5ab6-4fe5-8465-3aeda732105d service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:56:22.179802 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb7fb14b-5ab6-4fe5-8465-3aeda732105d service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:56:22.181437 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb7fb14b-5ab6-4fe5-8465-3aeda732105d service swift] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:56:22.190608 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1211/2429] 10.222.0.27 () {60 vars in 1384 bytes} [Sun Sep 11 00:56:22 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 11 00:56:23.978232 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0cfb15a-fa26-4fa8-97f4-41ed2ee02d14 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:56:23.988352 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0cfb15a-fa26-4fa8-97f4-41ed2ee02d14 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:56:23.998857 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ebd8c0f-33c5-477c-91b2-98ea08ceff1e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:56:23.999490 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ebd8c0f-33c5-477c-91b2-98ea08ceff1e service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:56:23.999765 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ebd8c0f-33c5-477c-91b2-98ea08ceff1e service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:56:24.000104 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ebd8c0f-33c5-477c-91b2-98ea08ceff1e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:56:24.064653 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ebd8c0f-33c5-477c-91b2-98ea08ceff1e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:56:24.066160 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ebd8c0f-33c5-477c-91b2-98ea08ceff1e service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:56:24.076110 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1219/2430] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:56:23 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 11 00:56:27.978452 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7fb14b-5ab6-4fe5-8465-3aeda732105d service swift] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:56:27.988134 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7fb14b-5ab6-4fe5-8465-3aeda732105d service swift] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:56:27.992434 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e960007-2537-422e-8079-be1728c8ec5c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:56:27.993223 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e960007-2537-422e-8079-be1728c8ec5c service neutron] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:56:27.993501 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e960007-2537-422e-8079-be1728c8ec5c service neutron] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:56:27.993777 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e960007-2537-422e-8079-be1728c8ec5c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:56:28.003612 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e960007-2537-422e-8079-be1728c8ec5c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:56:28.005042 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e960007-2537-422e-8079-be1728c8ec5c service neutron] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:56:28.015132 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1212/2431] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:56:27 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:56:34.071753 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=33 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:56:35.198992 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ada03b-89e8-4bfc-bc6c-fc848926a5c3 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:56:35.199578 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ada03b-89e8-4bfc-bc6c-fc848926a5c3 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:56:35.199578 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ada03b-89e8-4bfc-bc6c-fc848926a5c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:56:35.220934 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-93ada03b-89e8-4bfc-bc6c-fc848926a5c3 None None] MFA Rules not processed for user `d886b168a8eb4424987555c073d71727`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:56:35.222152 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-93ada03b-89e8-4bfc-bc6c-fc848926a5c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:56:35.276341 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1220/2432] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:56:35 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:56:38.010371 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=12 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:56:45.269044 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=25 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:57:25.573858 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1245b731-af58-484a-9a5f-66ccee90f4c1 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:57:25.574438 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1245b731-af58-484a-9a5f-66ccee90f4c1 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:57:25.574438 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1245b731-af58-484a-9a5f-66ccee90f4c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:57:25.594918 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-1245b731-af58-484a-9a5f-66ccee90f4c1 None None] MFA Rules not processed for user `73a3d3c48913474ab8fa8fedd1bc7d03`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:57:25.596113 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-1245b731-af58-484a-9a5f-66ccee90f4c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:57:25.651725 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1213/2433] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:57:25 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:57:29.045753 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ada03b-89e8-4bfc-bc6c-fc848926a5c3 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:57:29.049278 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-93ada03b-89e8-4bfc-bc6c-fc848926a5c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:57:29.150186 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ada03b-89e8-4bfc-bc6c-fc848926a5c3 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:57:29.155175 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f1c745c-f87f-447b-8c52-a8c5937d9df2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:57:29.155809 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1c745c-f87f-447b-8c52-a8c5937d9df2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:57:29.156089 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1c745c-f87f-447b-8c52-a8c5937d9df2 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:57:29.156358 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1c745c-f87f-447b-8c52-a8c5937d9df2 admin admin] PATH_INFO: `/v3/users/5df6979effd84ece8d75b631ba7da196` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:57:29.157477 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f1c745c-f87f-447b-8c52-a8c5937d9df2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5df6979effd84ece8d75b631ba7da196)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:57:29.159702 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f1c745c-f87f-447b-8c52-a8c5937d9df2 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:57:29.210277 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-2f1c745c-f87f-447b-8c52-a8c5937d9df2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5df6979effd84ece8d75b631ba7da196'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:57:29.211787 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-2f1c745c-f87f-447b-8c52-a8c5937d9df2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5df6979effd84ece8d75b631ba7da196'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:57:29.212405 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-2f1c745c-f87f-447b-8c52-a8c5937d9df2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5df6979effd84ece8d75b631ba7da196'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:57:29.219532 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-2f1c745c-f87f-447b-8c52-a8c5937d9df2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5df6979effd84ece8d75b631ba7da196'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:57:29.232131 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-2f1c745c-f87f-447b-8c52-a8c5937d9df2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5df6979effd84ece8d75b631ba7da196'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:57:29.246076 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1221/2434] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:57:29 2022] DELETE /identity/v3/users/5df6979effd84ece8d75b631ba7da196 => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:57:29.425122 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1245b731-af58-484a-9a5f-66ccee90f4c1 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:57:29.441125 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1245b731-af58-484a-9a5f-66ccee90f4c1 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:57:29.504204 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2bf8a6b-933a-405d-be69-ae2744c6dfb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:57:29.504863 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bf8a6b-933a-405d-be69-ae2744c6dfb2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:57:29.505164 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bf8a6b-933a-405d-be69-ae2744c6dfb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:57:29.505442 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bf8a6b-933a-405d-be69-ae2744c6dfb2 admin admin] PATH_INFO: `/v3/projects/6a58aefc25e141ca93821a80859875e8` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:57:29.506594 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2bf8a6b-933a-405d-be69-ae2744c6dfb2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6a58aefc25e141ca93821a80859875e8)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:57:29.508641 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2bf8a6b-933a-405d-be69-ae2744c6dfb2 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:57:29.561092 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-d2bf8a6b-933a-405d-be69-ae2744c6dfb2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6a58aefc25e141ca93821a80859875e8', 'request_id': 'req-d2bf8a6b-933a-405d-be69-ae2744c6dfb2'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:57:29.575029 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-d2bf8a6b-933a-405d-be69-ae2744c6dfb2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6a58aefc25e141ca93821a80859875e8', 'request_id': 'req-d2bf8a6b-933a-405d-be69-ae2744c6dfb2'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:57:29.584953 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-d2bf8a6b-933a-405d-be69-ae2744c6dfb2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6a58aefc25e141ca93821a80859875e8', 'request_id': 'req-d2bf8a6b-933a-405d-be69-ae2744c6dfb2'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:57:29.603456 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-d2bf8a6b-933a-405d-be69-ae2744c6dfb2 admin admin] The token cache is being invalidate because project 6a58aefc25e141ca93821a80859875e8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87923) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:57:29.604614 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-d2bf8a6b-933a-405d-be69-ae2744c6dfb2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:57:29.605495 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-d2bf8a6b-933a-405d-be69-ae2744c6dfb2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:57:29.610663 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1214/2435] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:57:29 2022] DELETE /identity/v3/projects/6a58aefc25e141ca93821a80859875e8 => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:57:29.612675 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9103927e-535a-4e01-8ac9-92e8ee3b34fc None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:57:29.613009 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9103927e-535a-4e01-8ac9-92e8ee3b34fc None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:57:29.613333 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9103927e-535a-4e01-8ac9-92e8ee3b34fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:57:29.637233 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-9103927e-535a-4e01-8ac9-92e8ee3b34fc None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:57:29.638554 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-9103927e-535a-4e01-8ac9-92e8ee3b34fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:57:29.700151 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1222/2436] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:57:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:57:29.707926 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5022b44e-a81a-4dc0-96ae-160bec96359d None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:57:29.708201 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5022b44e-a81a-4dc0-96ae-160bec96359d None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:57:29.708476 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5022b44e-a81a-4dc0-96ae-160bec96359d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:57:29.728281 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-5022b44e-a81a-4dc0-96ae-160bec96359d None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:57:29.729485 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-5022b44e-a81a-4dc0-96ae-160bec96359d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:57:29.737276 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1215/2437] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:57:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:57:29.742181 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9103927e-535a-4e01-8ac9-92e8ee3b34fc None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:57:29.756645 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9103927e-535a-4e01-8ac9-92e8ee3b34fc None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:57:29.761273 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4aab439-00d8-4888-bef4-45aba9183cbc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:57:29.761875 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4aab439-00d8-4888-bef4-45aba9183cbc admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:57:29.762151 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4aab439-00d8-4888-bef4-45aba9183cbc admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:57:29.762422 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4aab439-00d8-4888-bef4-45aba9183cbc admin admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:57:29.763104 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4aab439-00d8-4888-bef4-45aba9183cbc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:57:29.763413 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4aab439-00d8-4888-bef4-45aba9183cbc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:57:29.765011 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4aab439-00d8-4888-bef4-45aba9183cbc admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:57:29.777293 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1223/2438] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 00:57:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:57:29.781670 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5022b44e-a81a-4dc0-96ae-160bec96359d None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:57:29.791554 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5022b44e-a81a-4dc0-96ae-160bec96359d None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:57:29.796217 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e92353b-a9b4-4eaa-8394-e9a95cb84af3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:57:29.796818 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e92353b-a9b4-4eaa-8394-e9a95cb84af3 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:57:29.797094 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e92353b-a9b4-4eaa-8394-e9a95cb84af3 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:57:29.797360 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e92353b-a9b4-4eaa-8394-e9a95cb84af3 admin admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:57:29.797991 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e92353b-a9b4-4eaa-8394-e9a95cb84af3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:57:29.799907 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e92353b-a9b4-4eaa-8394-e9a95cb84af3 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:57:29.829367 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1216/2439] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:57:29 2022] POST /identity/v3/projects => generated 373 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:57:29.833476 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4aab439-00d8-4888-bef4-45aba9183cbc admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:57:29.843768 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4aab439-00d8-4888-bef4-45aba9183cbc admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:57:29.847482 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:57:29.905254 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9201bc8a-08a7-41ff-94e0-1708b69b172a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:57:29.905888 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9201bc8a-08a7-41ff-94e0-1708b69b172a admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:57:29.906167 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9201bc8a-08a7-41ff-94e0-1708b69b172a admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:57:29.906438 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9201bc8a-08a7-41ff-94e0-1708b69b172a admin admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:57:29.907098 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9201bc8a-08a7-41ff-94e0-1708b69b172a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:57:29.909278 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9201bc8a-08a7-41ff-94e0-1708b69b172a admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:57:29.934990 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1224/2440] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:57:29 2022] POST /identity/v3/users => generated 287 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:57:29.936072 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e92353b-a9b4-4eaa-8394-e9a95cb84af3 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:57:29.946740 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e92353b-a9b4-4eaa-8394-e9a95cb84af3 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:57:29.951369 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-072eb88f-da06-4555-bf5e-62c431690fdf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:57:29.951966 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072eb88f-da06-4555-bf5e-62c431690fdf admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:57:29.952242 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072eb88f-da06-4555-bf5e-62c431690fdf admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:57:29.952510 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072eb88f-da06-4555-bf5e-62c431690fdf admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:57:29.953149 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-072eb88f-da06-4555-bf5e-62c431690fdf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:57:29.954779 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-072eb88f-da06-4555-bf5e-62c431690fdf admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:57:29.956151 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:57:29.969606 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1217/2441] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:57:29 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:57:29.970865 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9201bc8a-08a7-41ff-94e0-1708b69b172a admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:57:29.982068 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9201bc8a-08a7-41ff-94e0-1708b69b172a admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:57:29.986842 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29846f05-e88c-423c-b4d1-310244b460b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:57:29.987439 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29846f05-e88c-423c-b4d1-310244b460b4 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:57:29.987710 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29846f05-e88c-423c-b4d1-310244b460b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:57:29.987986 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29846f05-e88c-423c-b4d1-310244b460b4 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:57:29.988626 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29846f05-e88c-423c-b4d1-310244b460b4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:57:29.990929 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29846f05-e88c-423c-b4d1-310244b460b4 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:57:30.001565 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1225/2442] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:57:29 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 11 00:57:30.005918 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-072eb88f-da06-4555-bf5e-62c431690fdf admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:57:30.016557 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-072eb88f-da06-4555-bf5e-62c431690fdf admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:57:30.022090 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce0b468-84d4-4f8a-bc4f-aa1b712c8e95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:57:30.022738 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce0b468-84d4-4f8a-bc4f-aa1b712c8e95 admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:57:30.023045 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce0b468-84d4-4f8a-bc4f-aa1b712c8e95 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:57:30.023312 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce0b468-84d4-4f8a-bc4f-aa1b712c8e95 admin admin] PATH_INFO: `/v3/projects/f4e85ba2dc0e4b628255dafdef5d7bcb/users/a66a2861b54c4aa79dd1b6f916287790/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:57:30.041242 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bce0b468-84d4-4f8a-bc4f-aa1b712c8e95 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f4e85ba2dc0e4b628255dafdef5d7bcb, user_id=a66a2861b54c4aa79dd1b6f916287790, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:57:30.044343 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bce0b468-84d4-4f8a-bc4f-aa1b712c8e95 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:57:30.058409 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1218/2443] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:57:30 2022] PUT /identity/v3/projects/f4e85ba2dc0e4b628255dafdef5d7bcb/users/a66a2861b54c4aa79dd1b6f916287790/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:57:30.068840 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05db892b-ca52-4cc3-b2fe-2a75a91aa548 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:57:30.069311 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05db892b-ca52-4cc3-b2fe-2a75a91aa548 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:57:30.069406 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05db892b-ca52-4cc3-b2fe-2a75a91aa548 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:57:30.097897 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-05db892b-ca52-4cc3-b2fe-2a75a91aa548 None None] MFA Rules not processed for user `a66a2861b54c4aa79dd1b6f916287790`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:57:30.100144 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-05db892b-ca52-4cc3-b2fe-2a75a91aa548 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:57:30.172611 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1226/2444] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:57:30 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:57:30.178390 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce0b468-84d4-4f8a-bc4f-aa1b712c8e95 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:57:30.248877 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce0b468-84d4-4f8a-bc4f-aa1b712c8e95 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:57:30.253606 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d4ee0df-004a-485b-aa6d-27c2f53836a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:57:30.254210 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d4ee0df-004a-485b-aa6d-27c2f53836a3 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:57:30.254488 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d4ee0df-004a-485b-aa6d-27c2f53836a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:57:30.254755 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d4ee0df-004a-485b-aa6d-27c2f53836a3 admin admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:57:30.255467 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d4ee0df-004a-485b-aa6d-27c2f53836a3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:57:30.257598 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d4ee0df-004a-485b-aa6d-27c2f53836a3 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:57:30.288067 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1219/2445] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:57:30 2022] POST /identity/v3/projects => generated 373 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:57:30.292209 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05db892b-ca52-4cc3-b2fe-2a75a91aa548 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:57:30.303087 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05db892b-ca52-4cc3-b2fe-2a75a91aa548 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:57:30.359852 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6348f0d7-59f7-4aa5-ba96-3179168f61a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:57:30.360460 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6348f0d7-59f7-4aa5-ba96-3179168f61a3 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:57:30.360742 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6348f0d7-59f7-4aa5-ba96-3179168f61a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:57:30.361016 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6348f0d7-59f7-4aa5-ba96-3179168f61a3 admin admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:57:30.361657 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6348f0d7-59f7-4aa5-ba96-3179168f61a3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:57:30.363613 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6348f0d7-59f7-4aa5-ba96-3179168f61a3 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:57:30.384234 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1227/2446] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:57:30 2022] POST /identity/v3/users => generated 293 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:57:30.388381 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d4ee0df-004a-485b-aa6d-27c2f53836a3 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:57:30.399229 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d4ee0df-004a-485b-aa6d-27c2f53836a3 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:57:30.403838 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea5d337-7435-439e-b0c7-696ebbaa7f27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:57:30.404448 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea5d337-7435-439e-b0c7-696ebbaa7f27 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:57:30.404723 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea5d337-7435-439e-b0c7-696ebbaa7f27 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:57:30.404987 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea5d337-7435-439e-b0c7-696ebbaa7f27 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:57:30.405645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fea5d337-7435-439e-b0c7-696ebbaa7f27 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:57:30.407425 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fea5d337-7435-439e-b0c7-696ebbaa7f27 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:57:30.419328 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1220/2447] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:57:30 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:57:30.423482 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6348f0d7-59f7-4aa5-ba96-3179168f61a3 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:57:30.429809 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:57:30.433996 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6348f0d7-59f7-4aa5-ba96-3179168f61a3 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:57:30.439580 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53b2e9d6-3d35-4813-8e9f-19f3d1272af8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:57:30.440222 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b2e9d6-3d35-4813-8e9f-19f3d1272af8 admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:57:30.440498 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b2e9d6-3d35-4813-8e9f-19f3d1272af8 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:57:30.440768 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b2e9d6-3d35-4813-8e9f-19f3d1272af8 admin admin] PATH_INFO: `/v3/projects/ac156806b2a7405a9abe3cacc2f564f9/users/3c9ac68bb0d042e182b842dff7a0cb7d/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:57:30.455369 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53b2e9d6-3d35-4813-8e9f-19f3d1272af8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ac156806b2a7405a9abe3cacc2f564f9, user_id=3c9ac68bb0d042e182b842dff7a0cb7d, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:57:30.458001 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53b2e9d6-3d35-4813-8e9f-19f3d1272af8 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:57:30.470933 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1228/2448] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:57:30 2022] PUT /identity/v3/projects/ac156806b2a7405a9abe3cacc2f564f9/users/3c9ac68bb0d042e182b842dff7a0cb7d/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:57:30.475885 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea5d337-7435-439e-b0c7-696ebbaa7f27 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:57:30.485889 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea5d337-7435-439e-b0c7-696ebbaa7f27 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:57:30.541928 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f9fdc2-7ab8-4fa4-b06b-780288e21692 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:57:30.542525 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f9fdc2-7ab8-4fa4-b06b-780288e21692 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:57:30.542851 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f9fdc2-7ab8-4fa4-b06b-780288e21692 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:57:30.543126 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f9fdc2-7ab8-4fa4-b06b-780288e21692 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:57:30.543760 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0f9fdc2-7ab8-4fa4-b06b-780288e21692 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:57:30.545355 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0f9fdc2-7ab8-4fa4-b06b-780288e21692 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:57:30.555360 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1221/2449] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:57:30 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:57:30.559522 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53b2e9d6-3d35-4813-8e9f-19f3d1272af8 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:57:30.569956 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53b2e9d6-3d35-4813-8e9f-19f3d1272af8 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:57:30.574588 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e515b192-77c3-43aa-80c2-b6b4517636b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:57:30.575260 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e515b192-77c3-43aa-80c2-b6b4517636b8 admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:57:30.575542 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e515b192-77c3-43aa-80c2-b6b4517636b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:57:30.575816 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e515b192-77c3-43aa-80c2-b6b4517636b8 admin admin] PATH_INFO: `/v3/projects/ac156806b2a7405a9abe3cacc2f564f9/users/3c9ac68bb0d042e182b842dff7a0cb7d/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:57:30.577714 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e515b192-77c3-43aa-80c2-b6b4517636b8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ac156806b2a7405a9abe3cacc2f564f9, user_id=3c9ac68bb0d042e182b842dff7a0cb7d, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:57:30.580387 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e515b192-77c3-43aa-80c2-b6b4517636b8 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:57:30.591130 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1229/2450] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:57:30 2022] PUT /identity/v3/projects/ac156806b2a7405a9abe3cacc2f564f9/users/3c9ac68bb0d042e182b842dff7a0cb7d/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:57:30.602889 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d17a5c-448b-4dec-a489-a31bf6db8171 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:57:30.603180 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d17a5c-448b-4dec-a489-a31bf6db8171 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:57:30.603453 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d17a5c-448b-4dec-a489-a31bf6db8171 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:57:30.633695 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-e9d17a5c-448b-4dec-a489-a31bf6db8171 None None] MFA Rules not processed for user `3c9ac68bb0d042e182b842dff7a0cb7d`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:57:30.634916 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-e9d17a5c-448b-4dec-a489-a31bf6db8171 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:57:30.664872 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:57:30.702735 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1222/2451] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:57:30 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:57:30.714878 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e515b192-77c3-43aa-80c2-b6b4517636b8 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:57:30.716619 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-e515b192-77c3-43aa-80c2-b6b4517636b8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:57:30.774853 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e515b192-77c3-43aa-80c2-b6b4517636b8 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:57:30.778447 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b79cd32e-4a2d-4ffc-9721-46cd4cbd0e2b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:57:30.779071 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b79cd32e-4a2d-4ffc-9721-46cd4cbd0e2b service cinder] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:57:30.779346 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b79cd32e-4a2d-4ffc-9721-46cd4cbd0e2b service cinder] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:57:30.779613 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b79cd32e-4a2d-4ffc-9721-46cd4cbd0e2b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:57:30.788684 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b79cd32e-4a2d-4ffc-9721-46cd4cbd0e2b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:57:30.790100 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b79cd32e-4a2d-4ffc-9721-46cd4cbd0e2b service cinder] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:57:30.799225 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1230/2452] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:57:30 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:57:30.875631 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9d17a5c-448b-4dec-a489-a31bf6db8171 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:57:30.885492 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9d17a5c-448b-4dec-a489-a31bf6db8171 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:57:30.889079 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d5ca98-6cd9-4ccb-9bec-8d8711bedb19 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:57:30.889666 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d5ca98-6cd9-4ccb-9bec-8d8711bedb19 service cinder] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:57:30.889937 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d5ca98-6cd9-4ccb-9bec-8d8711bedb19 service cinder] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:57:30.890202 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d5ca98-6cd9-4ccb-9bec-8d8711bedb19 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:57:30.946487 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36d5ca98-6cd9-4ccb-9bec-8d8711bedb19 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:57:30.947974 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36d5ca98-6cd9-4ccb-9bec-8d8711bedb19 service cinder] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:57:30.957789 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1223/2453] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:57:30 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:57:36.135633 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b79cd32e-4a2d-4ffc-9721-46cd4cbd0e2b service cinder] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:57:36.137017 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-b79cd32e-4a2d-4ffc-9721-46cd4cbd0e2b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:57:36.192369 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b79cd32e-4a2d-4ffc-9721-46cd4cbd0e2b service cinder] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:57:36.196546 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d808047d-63fe-4b93-a437-aa96ac19170d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:57:36.197140 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d808047d-63fe-4b93-a437-aa96ac19170d service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:57:36.197421 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d808047d-63fe-4b93-a437-aa96ac19170d service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:57:36.197687 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d808047d-63fe-4b93-a437-aa96ac19170d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:57:36.199410 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-d808047d-63fe-4b93-a437-aa96ac19170d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:57:36.269011 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d808047d-63fe-4b93-a437-aa96ac19170d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:57:36.270361 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d808047d-63fe-4b93-a437-aa96ac19170d service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:57:36.281226 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1231/2454] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:57:36 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:57:37.465280 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d5ca98-6cd9-4ccb-9bec-8d8711bedb19 service cinder] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:57:37.474743 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d5ca98-6cd9-4ccb-9bec-8d8711bedb19 service cinder] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:57:37.478467 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f80e8589-b7db-4190-b88e-1b32057e5802 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:57:37.479126 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80e8589-b7db-4190-b88e-1b32057e5802 service neutron] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:57:37.479388 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80e8589-b7db-4190-b88e-1b32057e5802 service neutron] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:57:37.479687 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80e8589-b7db-4190-b88e-1b32057e5802 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:57:37.481397 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-f80e8589-b7db-4190-b88e-1b32057e5802 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:57:37.496082 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f80e8589-b7db-4190-b88e-1b32057e5802 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:57:37.497728 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f80e8589-b7db-4190-b88e-1b32057e5802 service neutron] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:57:37.513866 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1224/2455] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:57:37 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:57:46.276354 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=93,keystone:INSERT=2 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:57:47.505265 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=49 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:57:48.640639 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52846ee1-ab0d-472f-a377-bc2f6015c142 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:57:48.641175 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52846ee1-ab0d-472f-a377-bc2f6015c142 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:57:48.641350 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52846ee1-ab0d-472f-a377-bc2f6015c142 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:57:48.662425 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-52846ee1-ab0d-472f-a377-bc2f6015c142 None None] MFA Rules not processed for user `d886b168a8eb4424987555c073d71727`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:57:48.664347 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-52846ee1-ab0d-472f-a377-bc2f6015c142 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:57:48.718337 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1232/2456] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:57:48 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:57:49.394810 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f80e8589-b7db-4190-b88e-1b32057e5802 service neutron] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:57:49.398136 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-f80e8589-b7db-4190-b88e-1b32057e5802 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:57:49.506944 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f80e8589-b7db-4190-b88e-1b32057e5802 service neutron] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:57:49.512944 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d8a46f-755c-474d-8409-50d13f1e977a service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cdbfc2ccd73e4a5eb8c999be5b094619', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:57:49.513562 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d8a46f-755c-474d-8409-50d13f1e977a service placement] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:57:49.513840 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d8a46f-755c-474d-8409-50d13f1e977a service placement] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:57:49.514109 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d8a46f-755c-474d-8409-50d13f1e977a service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:57:49.515890 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-88d8a46f-755c-474d-8409-50d13f1e977a service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:57:49.531483 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88d8a46f-755c-474d-8409-50d13f1e977a service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:57:49.534009 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88d8a46f-755c-474d-8409-50d13f1e977a service placement] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:57:49.545062 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1225/2457] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:57:49 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:57:58.711473 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=25 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:57:59.540267 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=32 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:58:00.546998 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52846ee1-ab0d-472f-a377-bc2f6015c142 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:00.549565 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-52846ee1-ab0d-472f-a377-bc2f6015c142 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:00.565236 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d8a46f-755c-474d-8409-50d13f1e977a service placement] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:00.570742 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52846ee1-ab0d-472f-a377-bc2f6015c142 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:00.574481 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e050436a-06e7-4575-8378-c52c8a772f94 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:00.575126 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e050436a-06e7-4575-8378-c52c8a772f94 service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:00.575406 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e050436a-06e7-4575-8378-c52c8a772f94 service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:00.575845 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e050436a-06e7-4575-8378-c52c8a772f94 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:00.577632 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-e050436a-06e7-4575-8378-c52c8a772f94 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:00.579057 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d8a46f-755c-474d-8409-50d13f1e977a service placement] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:00.583618 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb064faa-2ac9-4db0-8f13-59e766c9d00d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:00.584321 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb064faa-2ac9-4db0-8f13-59e766c9d00d service neutron] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:00.585411 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb064faa-2ac9-4db0-8f13-59e766c9d00d service neutron] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:00.586423 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb064faa-2ac9-4db0-8f13-59e766c9d00d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:00.590226 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e050436a-06e7-4575-8378-c52c8a772f94 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:00.594925 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e050436a-06e7-4575-8378-c52c8a772f94 service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:00.599303 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb064faa-2ac9-4db0-8f13-59e766c9d00d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:00.601041 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb064faa-2ac9-4db0-8f13-59e766c9d00d service neutron] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:00.609252 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1233/2458] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:58:00 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 63 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:58:00.611856 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1226/2459] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:58:00 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:58:01.493741 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e050436a-06e7-4575-8378-c52c8a772f94 service neutron] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:01.495171 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-e050436a-06e7-4575-8378-c52c8a772f94 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:01.510810 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e050436a-06e7-4575-8378-c52c8a772f94 service neutron] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:01.516176 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b2595d-a7c2-47eb-afe4-78fa06e2a31b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '835def0e6c6d4365a2767051509eb0d6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'ResellerAdmin', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:01.516770 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b2595d-a7c2-47eb-afe4-78fa06e2a31b service nova] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:01.517046 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b2595d-a7c2-47eb-afe4-78fa06e2a31b service nova] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:01.517316 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b2595d-a7c2-47eb-afe4-78fa06e2a31b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:01.519022 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-97b2595d-a7c2-47eb-afe4-78fa06e2a31b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:01.529713 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b2595d-a7c2-47eb-afe4-78fa06e2a31b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:01.531307 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b2595d-a7c2-47eb-afe4-78fa06e2a31b service nova] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:01.540906 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1234/2460] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:58:01 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:58:09.546980 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c750d2d-e29f-4dcf-baf0-40dfaac0d775 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:09.547535 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c750d2d-e29f-4dcf-baf0-40dfaac0d775 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:09.547654 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c750d2d-e29f-4dcf-baf0-40dfaac0d775 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:09.557927 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15dad7c7-ea5d-4c12-8fde-d384e98674e5 None None] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:09.558461 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15dad7c7-ea5d-4c12-8fde-d384e98674e5 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:09.558562 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15dad7c7-ea5d-4c12-8fde-d384e98674e5 None None] PATH_INFO: `/` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:09.560173 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1235/2461] 10.222.0.27 () {54 vars in 921 bytes} [Sun Sep 11 00:58:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 11 00:58:09.571573 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7780237-9b22-4553-a58c-d2f066f31f2f None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:09.572422 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7780237-9b22-4553-a58c-d2f066f31f2f None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:09.572996 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7780237-9b22-4553-a58c-d2f066f31f2f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:09.576282 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-4c750d2d-e29f-4dcf-baf0-40dfaac0d775 None None] MFA Rules not processed for user `9fc4274836844ae4ad5d16d74c36f671`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:58:09.577487 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-4c750d2d-e29f-4dcf-baf0-40dfaac0d775 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:09.599514 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-d7780237-9b22-4553-a58c-d2f066f31f2f None None] MFA Rules not processed for user `a5f7b13283cb4ebfb25160aa6ad0e2f4`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:58:09.601660 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-d7780237-9b22-4553-a58c-d2f066f31f2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:09.610408 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1236/2462] 10.222.0.27 () {58 vars in 1018 bytes} [Sun Sep 11 00:58:09 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:58:09.624106 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7780237-9b22-4553-a58c-d2f066f31f2f None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:09.633639 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7780237-9b22-4553-a58c-d2f066f31f2f None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:09.637860 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e61d9e5-823d-43b5-a731-a0eb3ca1eb2d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:09.638460 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e61d9e5-823d-43b5-a731-a0eb3ca1eb2d service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:09.638736 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e61d9e5-823d-43b5-a731-a0eb3ca1eb2d service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:09.639035 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e61d9e5-823d-43b5-a731-a0eb3ca1eb2d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:09.646362 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e61d9e5-823d-43b5-a731-a0eb3ca1eb2d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:09.647852 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e61d9e5-823d-43b5-a731-a0eb3ca1eb2d service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:09.656853 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1237/2463] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:58:09 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 33 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:58:09.657194 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1227/2464] 10.222.0.27 () {58 vars in 967 bytes} [Sun Sep 11 00:58:09 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:58:09.721660 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e61d9e5-823d-43b5-a731-a0eb3ca1eb2d service neutron] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:09.723150 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-3e61d9e5-823d-43b5-a731-a0eb3ca1eb2d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:09.747398 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361da1c7-a3ef-4119-9cdd-f7aa450bbbc1 None None] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:09.747582 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361da1c7-a3ef-4119-9cdd-f7aa450bbbc1 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:09.748312 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361da1c7-a3ef-4119-9cdd-f7aa450bbbc1 None None] PATH_INFO: `/` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:09.750221 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1228/2465] 10.222.0.27 () {54 vars in 921 bytes} [Sun Sep 11 00:58:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 11 00:58:09.755938 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298dfa92-1aa4-453f-8d3a-e5cf1d5409ff None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:09.756298 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298dfa92-1aa4-453f-8d3a-e5cf1d5409ff None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:09.756636 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298dfa92-1aa4-453f-8d3a-e5cf1d5409ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:09.782123 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-298dfa92-1aa4-453f-8d3a-e5cf1d5409ff None None] MFA Rules not processed for user `835def0e6c6d4365a2767051509eb0d6`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:58:09.783631 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-298dfa92-1aa4-453f-8d3a-e5cf1d5409ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:09.797814 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1229/2466] 10.222.0.27 () {58 vars in 1018 bytes} [Sun Sep 11 00:58:09 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:58:09.809858 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-298dfa92-1aa4-453f-8d3a-e5cf1d5409ff None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:09.824933 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-298dfa92-1aa4-453f-8d3a-e5cf1d5409ff None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:09.829511 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3407e1e-7ebb-4bdc-8307-92d516d594f3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:09.830034 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e61d9e5-823d-43b5-a731-a0eb3ca1eb2d service neutron] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:09.830164 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3407e1e-7ebb-4bdc-8307-92d516d594f3 service neutron] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:09.830376 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3407e1e-7ebb-4bdc-8307-92d516d594f3 service neutron] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:09.830978 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3407e1e-7ebb-4bdc-8307-92d516d594f3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:09.837371 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71322aec-267b-4017-a51c-fde15db9624f service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bcb73dec73464a3abc4b774abb4f416f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:09.838056 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71322aec-267b-4017-a51c-fde15db9624f service swift] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:09.838645 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71322aec-267b-4017-a51c-fde15db9624f service swift] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:09.839121 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71322aec-267b-4017-a51c-fde15db9624f service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:09.846200 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3407e1e-7ebb-4bdc-8307-92d516d594f3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:09.848275 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3407e1e-7ebb-4bdc-8307-92d516d594f3 service neutron] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:09.851004 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71322aec-267b-4017-a51c-fde15db9624f service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:09.853317 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71322aec-267b-4017-a51c-fde15db9624f service swift] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:09.863260 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1238/2467] 10.222.0.27 () {60 vars in 1384 bytes} [Sun Sep 11 00:58:09 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 11 00:58:09.866050 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1230/2468] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:58:09 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 57 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:58:10.365111 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3407e1e-7ebb-4bdc-8307-92d516d594f3 service neutron] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:10.451910 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3407e1e-7ebb-4bdc-8307-92d516d594f3 service neutron] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:10.457268 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f42f3d9-c3e5-4839-af3a-57685d2e6433 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:10.457954 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f42f3d9-c3e5-4839-af3a-57685d2e6433 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:10.458218 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f42f3d9-c3e5-4839-af3a-57685d2e6433 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:10.458517 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f42f3d9-c3e5-4839-af3a-57685d2e6433 admin admin] PATH_INFO: `/v3/users/a66a2861b54c4aa79dd1b6f916287790` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:10.459799 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f42f3d9-c3e5-4839-af3a-57685d2e6433 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a66a2861b54c4aa79dd1b6f916287790)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:10.463578 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f42f3d9-c3e5-4839-af3a-57685d2e6433 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:10.517899 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-8f42f3d9-c3e5-4839-af3a-57685d2e6433 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a66a2861b54c4aa79dd1b6f916287790'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:10.518467 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-8f42f3d9-c3e5-4839-af3a-57685d2e6433 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a66a2861b54c4aa79dd1b6f916287790'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:10.519098 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-8f42f3d9-c3e5-4839-af3a-57685d2e6433 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a66a2861b54c4aa79dd1b6f916287790'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:10.527893 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-8f42f3d9-c3e5-4839-af3a-57685d2e6433 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a66a2861b54c4aa79dd1b6f916287790'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:10.544860 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-8f42f3d9-c3e5-4839-af3a-57685d2e6433 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a66a2861b54c4aa79dd1b6f916287790'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:10.546964 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=91,keystone:DELETE=9 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:58:10.563216 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1231/2469] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:58:10 2022] DELETE /identity/v3/users/a66a2861b54c4aa79dd1b6f916287790 => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:58:10.568092 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c59d72-11d3-442f-bf40-f36fd29b741b None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:10.568532 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c59d72-11d3-442f-bf40-f36fd29b741b None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:10.568711 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c59d72-11d3-442f-bf40-f36fd29b741b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:10.591316 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-a3c59d72-11d3-442f-bf40-f36fd29b741b None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:58:10.592719 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-a3c59d72-11d3-442f-bf40-f36fd29b741b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:10.659547 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1239/2470] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:58:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:58:10.669973 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f42f3d9-c3e5-4839-af3a-57685d2e6433 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:10.671558 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-8f42f3d9-c3e5-4839-af3a-57685d2e6433 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:10.738041 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f42f3d9-c3e5-4839-af3a-57685d2e6433 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:10.742907 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-006ea4ce-4b4f-4103-8577-dbc9f078b3b9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:10.743766 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-006ea4ce-4b4f-4103-8577-dbc9f078b3b9 service neutron] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:10.744233 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-006ea4ce-4b4f-4103-8577-dbc9f078b3b9 service neutron] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:10.744720 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-006ea4ce-4b4f-4103-8577-dbc9f078b3b9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:10.755468 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-006ea4ce-4b4f-4103-8577-dbc9f078b3b9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:10.758148 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-006ea4ce-4b4f-4103-8577-dbc9f078b3b9 service neutron] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:10.769181 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1232/2471] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:58:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:58:11.061533 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3c59d72-11d3-442f-bf40-f36fd29b741b None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:11.064070 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-a3c59d72-11d3-442f-bf40-f36fd29b741b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:11.081111 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3c59d72-11d3-442f-bf40-f36fd29b741b None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:11.088182 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e85c17-c5d7-4ee2-be05-92484c4050dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:11.089505 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e85c17-c5d7-4ee2-be05-92484c4050dd admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:11.090058 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e85c17-c5d7-4ee2-be05-92484c4050dd admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:11.090765 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e85c17-c5d7-4ee2-be05-92484c4050dd admin admin] PATH_INFO: `/v3/projects/f4e85ba2dc0e4b628255dafdef5d7bcb` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:11.092279 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2e85c17-c5d7-4ee2-be05-92484c4050dd admin admin] RBAC: Authorizing `identity:delete_project(project_id=f4e85ba2dc0e4b628255dafdef5d7bcb)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:11.096055 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2e85c17-c5d7-4ee2-be05-92484c4050dd admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:11.159343 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:58:11.167516 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-b2e85c17-c5d7-4ee2-be05-92484c4050dd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f4e85ba2dc0e4b628255dafdef5d7bcb', 'request_id': 'req-b2e85c17-c5d7-4ee2-be05-92484c4050dd'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:11.172828 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-006ea4ce-4b4f-4103-8577-dbc9f078b3b9 service neutron] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:11.174758 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-006ea4ce-4b4f-4103-8577-dbc9f078b3b9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:11.181213 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-b2e85c17-c5d7-4ee2-be05-92484c4050dd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f4e85ba2dc0e4b628255dafdef5d7bcb', 'request_id': 'req-b2e85c17-c5d7-4ee2-be05-92484c4050dd'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:11.198295 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-b2e85c17-c5d7-4ee2-be05-92484c4050dd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f4e85ba2dc0e4b628255dafdef5d7bcb', 'request_id': 'req-b2e85c17-c5d7-4ee2-be05-92484c4050dd'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:11.216129 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-b2e85c17-c5d7-4ee2-be05-92484c4050dd admin admin] The token cache is being invalidate because project f4e85ba2dc0e4b628255dafdef5d7bcb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87924) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:58:11.216287 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-b2e85c17-c5d7-4ee2-be05-92484c4050dd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:11.217558 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-b2e85c17-c5d7-4ee2-be05-92484c4050dd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:11.220140 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1240/2472] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:58:11 2022] DELETE /identity/v3/projects/f4e85ba2dc0e4b628255dafdef5d7bcb => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:58:11.223281 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e85c17-c5d7-4ee2-be05-92484c4050dd admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:11.224883 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-b2e85c17-c5d7-4ee2-be05-92484c4050dd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:11.309564 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e85c17-c5d7-4ee2-be05-92484c4050dd admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:11.315135 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16097b05-af40-4fd6-b648-61dcdcd7e185 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:11.316154 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16097b05-af40-4fd6-b648-61dcdcd7e185 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:11.316667 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16097b05-af40-4fd6-b648-61dcdcd7e185 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:11.317163 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16097b05-af40-4fd6-b648-61dcdcd7e185 admin admin] PATH_INFO: `/v3/users/3c9ac68bb0d042e182b842dff7a0cb7d` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:11.318601 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16097b05-af40-4fd6-b648-61dcdcd7e185 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3c9ac68bb0d042e182b842dff7a0cb7d)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:11.322595 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16097b05-af40-4fd6-b648-61dcdcd7e185 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:11.332007 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-006ea4ce-4b4f-4103-8577-dbc9f078b3b9 service neutron] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:11.390025 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-16097b05-af40-4fd6-b648-61dcdcd7e185 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3c9ac68bb0d042e182b842dff7a0cb7d'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:11.390902 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-16097b05-af40-4fd6-b648-61dcdcd7e185 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3c9ac68bb0d042e182b842dff7a0cb7d'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:11.391576 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-16097b05-af40-4fd6-b648-61dcdcd7e185 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3c9ac68bb0d042e182b842dff7a0cb7d'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:11.403480 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-16097b05-af40-4fd6-b648-61dcdcd7e185 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3c9ac68bb0d042e182b842dff7a0cb7d'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:11.439857 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-16097b05-af40-4fd6-b648-61dcdcd7e185 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3c9ac68bb0d042e182b842dff7a0cb7d'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:11.450360 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4421bebd-b1a3-4f46-80d4-06bf54b74203 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '835def0e6c6d4365a2767051509eb0d6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'ResellerAdmin', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:11.451207 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4421bebd-b1a3-4f46-80d4-06bf54b74203 service nova] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:11.451474 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4421bebd-b1a3-4f46-80d4-06bf54b74203 service nova] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:11.451896 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4421bebd-b1a3-4f46-80d4-06bf54b74203 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:11.454090 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-4421bebd-b1a3-4f46-80d4-06bf54b74203 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:11.458878 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1241/2473] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:58:11 2022] DELETE /identity/v3/users/3c9ac68bb0d042e182b842dff7a0cb7d => generated 0 bytes in 236 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:58:11.483044 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4421bebd-b1a3-4f46-80d4-06bf54b74203 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:11.484850 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4421bebd-b1a3-4f46-80d4-06bf54b74203 service nova] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:11.498305 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1233/2474] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:58:11 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 326 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 11 00:58:11.864263 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16097b05-af40-4fd6-b648-61dcdcd7e185 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:11.866993 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-16097b05-af40-4fd6-b648-61dcdcd7e185 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:11.933372 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:58:11.961695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16097b05-af40-4fd6-b648-61dcdcd7e185 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:11.966588 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16ecb2f1-38bd-4338-b96e-bd5c71459b58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:11.967256 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ecb2f1-38bd-4338-b96e-bd5c71459b58 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:11.967770 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ecb2f1-38bd-4338-b96e-bd5c71459b58 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:11.968039 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ecb2f1-38bd-4338-b96e-bd5c71459b58 admin admin] PATH_INFO: `/v3/projects/ac156806b2a7405a9abe3cacc2f564f9` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:11.969176 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16ecb2f1-38bd-4338-b96e-bd5c71459b58 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ac156806b2a7405a9abe3cacc2f564f9)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:11.971126 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16ecb2f1-38bd-4338-b96e-bd5c71459b58 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:12.029497 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-16ecb2f1-38bd-4338-b96e-bd5c71459b58 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ac156806b2a7405a9abe3cacc2f564f9', 'request_id': 'req-16ecb2f1-38bd-4338-b96e-bd5c71459b58'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:12.036580 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-16ecb2f1-38bd-4338-b96e-bd5c71459b58 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ac156806b2a7405a9abe3cacc2f564f9', 'request_id': 'req-16ecb2f1-38bd-4338-b96e-bd5c71459b58'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:12.045592 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-16ecb2f1-38bd-4338-b96e-bd5c71459b58 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ac156806b2a7405a9abe3cacc2f564f9', 'request_id': 'req-16ecb2f1-38bd-4338-b96e-bd5c71459b58'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:12.058937 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-16ecb2f1-38bd-4338-b96e-bd5c71459b58 admin admin] The token cache is being invalidate because project ac156806b2a7405a9abe3cacc2f564f9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87924) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:58:12.059242 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-16ecb2f1-38bd-4338-b96e-bd5c71459b58 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:12.059993 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-16ecb2f1-38bd-4338-b96e-bd5c71459b58 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:12.062217 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1242/2475] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:58:11 2022] DELETE /identity/v3/projects/ac156806b2a7405a9abe3cacc2f564f9 => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:58:12.067360 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-101980b7-3914-4c0d-b416-f2fc7bf1401f None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:12.067847 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-101980b7-3914-4c0d-b416-f2fc7bf1401f None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:12.067952 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-101980b7-3914-4c0d-b416-f2fc7bf1401f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:12.093486 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-101980b7-3914-4c0d-b416-f2fc7bf1401f None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:58:12.094674 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-101980b7-3914-4c0d-b416-f2fc7bf1401f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:12.122396 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:INSERT=1,keystone:DELETE=1,keystone:SELECT=98 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:58:12.191077 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1234/2476] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:58:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:58:12.202743 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03cbdd2a-0e4f-4234-bdbf-2dfa0242af59 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:12.203003 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03cbdd2a-0e4f-4234-bdbf-2dfa0242af59 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:12.203313 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03cbdd2a-0e4f-4234-bdbf-2dfa0242af59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:12.236046 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-03cbdd2a-0e4f-4234-bdbf-2dfa0242af59 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:58:12.237414 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-03cbdd2a-0e4f-4234-bdbf-2dfa0242af59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:12.255885 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1243/2477] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:58:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 54 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:58:12.256646 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-101980b7-3914-4c0d-b416-f2fc7bf1401f None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:12.267750 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-101980b7-3914-4c0d-b416-f2fc7bf1401f None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:12.273865 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f6ce13-ed7f-4629-8bc5-0447c106ae40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:12.274322 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f6ce13-ed7f-4629-8bc5-0447c106ae40 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:12.274681 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f6ce13-ed7f-4629-8bc5-0447c106ae40 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:12.275124 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f6ce13-ed7f-4629-8bc5-0447c106ae40 admin admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:12.276447 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25f6ce13-ed7f-4629-8bc5-0447c106ae40 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:58:12.276790 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25f6ce13-ed7f-4629-8bc5-0447c106ae40 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:12.278754 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25f6ce13-ed7f-4629-8bc5-0447c106ae40 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:12.323981 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1235/2478] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 00:58:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 68 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:58:12.329591 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03cbdd2a-0e4f-4234-bdbf-2dfa0242af59 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:12.342370 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03cbdd2a-0e4f-4234-bdbf-2dfa0242af59 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:12.347923 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-587daa67-4f3b-4e92-8e88-c2394645b1c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:12.348935 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587daa67-4f3b-4e92-8e88-c2394645b1c1 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:12.349317 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587daa67-4f3b-4e92-8e88-c2394645b1c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:12.349707 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587daa67-4f3b-4e92-8e88-c2394645b1c1 admin admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:12.350546 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-587daa67-4f3b-4e92-8e88-c2394645b1c1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:12.355032 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-587daa67-4f3b-4e92-8e88-c2394645b1c1 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:12.458867 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f6ce13-ed7f-4629-8bc5-0447c106ae40 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:12.459686 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1244/2479] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:58:12 2022] POST /identity/v3/projects => generated 379 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:58:12.473278 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f6ce13-ed7f-4629-8bc5-0447c106ae40 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:12.593109 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57957ac7-2481-48f9-b020-3a45f567cc79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:12.593998 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57957ac7-2481-48f9-b020-3a45f567cc79 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:12.595095 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57957ac7-2481-48f9-b020-3a45f567cc79 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:12.595646 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57957ac7-2481-48f9-b020-3a45f567cc79 admin admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:12.598620 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57957ac7-2481-48f9-b020-3a45f567cc79 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:12.601857 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57957ac7-2481-48f9-b020-3a45f567cc79 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:12.630488 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1236/2480] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:58:12 2022] POST /identity/v3/users => generated 290 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:58:12.632870 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-587daa67-4f3b-4e92-8e88-c2394645b1c1 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:12.644972 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-587daa67-4f3b-4e92-8e88-c2394645b1c1 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:12.650659 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f84b19-e65f-4d43-a16a-09597701c62d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:12.651451 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f84b19-e65f-4d43-a16a-09597701c62d admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:12.651890 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f84b19-e65f-4d43-a16a-09597701c62d admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:12.652170 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f84b19-e65f-4d43-a16a-09597701c62d admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:12.652824 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04f84b19-e65f-4d43-a16a-09597701c62d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:12.654443 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04f84b19-e65f-4d43-a16a-09597701c62d admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:12.669554 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1245/2481] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:58:12 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:58:12.670461 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57957ac7-2481-48f9-b020-3a45f567cc79 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:12.681100 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57957ac7-2481-48f9-b020-3a45f567cc79 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:12.686585 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57a7dbfb-3d20-4591-816b-d70d3d13e2df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:12.687379 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a7dbfb-3d20-4591-816b-d70d3d13e2df admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:12.687497 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a7dbfb-3d20-4591-816b-d70d3d13e2df admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:12.687757 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a7dbfb-3d20-4591-816b-d70d3d13e2df admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:12.688403 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57a7dbfb-3d20-4591-816b-d70d3d13e2df admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:12.690013 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57a7dbfb-3d20-4591-816b-d70d3d13e2df admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:12.701184 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1237/2482] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:58:12 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 11 00:58:12.705200 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f84b19-e65f-4d43-a16a-09597701c62d admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:12.718121 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f84b19-e65f-4d43-a16a-09597701c62d admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:12.723399 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74eded50-3c6f-42ff-9608-f934ab9511e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:12.724066 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74eded50-3c6f-42ff-9608-f934ab9511e3 admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:12.724561 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74eded50-3c6f-42ff-9608-f934ab9511e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:12.724690 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74eded50-3c6f-42ff-9608-f934ab9511e3 admin admin] PATH_INFO: `/v3/projects/655a706eb2644a7abb0740a38248f6ac/users/50a714abb86f42a19c6dc860a489d6d3/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:12.741961 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74eded50-3c6f-42ff-9608-f934ab9511e3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=655a706eb2644a7abb0740a38248f6ac, user_id=50a714abb86f42a19c6dc860a489d6d3, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:12.744888 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74eded50-3c6f-42ff-9608-f934ab9511e3 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:12.766879 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1246/2483] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:58:12 2022] PUT /identity/v3/projects/655a706eb2644a7abb0740a38248f6ac/users/50a714abb86f42a19c6dc860a489d6d3/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:58:12.773735 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e08bf845-132b-4d80-9f60-74c2a0f7a702 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:12.774711 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e08bf845-132b-4d80-9f60-74c2a0f7a702 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:12.775145 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e08bf845-132b-4d80-9f60-74c2a0f7a702 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:12.810612 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-e08bf845-132b-4d80-9f60-74c2a0f7a702 None None] MFA Rules not processed for user `50a714abb86f42a19c6dc860a489d6d3`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:58:12.811877 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-e08bf845-132b-4d80-9f60-74c2a0f7a702 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:12.886253 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1238/2484] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:58:12 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:58:12.890532 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74eded50-3c6f-42ff-9608-f934ab9511e3 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:12.972225 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74eded50-3c6f-42ff-9608-f934ab9511e3 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:12.977812 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff22ba36-a6fe-456b-8e74-e48cc0309b8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:12.978630 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff22ba36-a6fe-456b-8e74-e48cc0309b8d admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:12.979148 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff22ba36-a6fe-456b-8e74-e48cc0309b8d admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:12.979287 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff22ba36-a6fe-456b-8e74-e48cc0309b8d admin admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:12.979891 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff22ba36-a6fe-456b-8e74-e48cc0309b8d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:12.981789 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff22ba36-a6fe-456b-8e74-e48cc0309b8d admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:13.003076 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:58:13.015601 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1247/2485] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:58:12 2022] POST /identity/v3/projects => generated 381 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:58:13.017321 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e08bf845-132b-4d80-9f60-74c2a0f7a702 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:13.028799 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e08bf845-132b-4d80-9f60-74c2a0f7a702 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:13.038010 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:58:13.093997 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b9f7211-2aa7-4766-a7b1-609176484a05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:13.094743 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b9f7211-2aa7-4766-a7b1-609176484a05 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:13.094949 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b9f7211-2aa7-4766-a7b1-609176484a05 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:13.095221 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b9f7211-2aa7-4766-a7b1-609176484a05 admin admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:13.095864 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b9f7211-2aa7-4766-a7b1-609176484a05 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:13.097817 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b9f7211-2aa7-4766-a7b1-609176484a05 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:13.123548 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1239/2486] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:58:13 2022] POST /identity/v3/users => generated 297 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:58:13.127628 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff22ba36-a6fe-456b-8e74-e48cc0309b8d admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:13.139022 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff22ba36-a6fe-456b-8e74-e48cc0309b8d admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:13.143816 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa7543f-d134-4763-b418-0173c13c23ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:13.144450 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa7543f-d134-4763-b418-0173c13c23ee admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:13.144739 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa7543f-d134-4763-b418-0173c13c23ee admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:13.145020 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa7543f-d134-4763-b418-0173c13c23ee admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:13.145658 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baa7543f-d134-4763-b418-0173c13c23ee admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:13.147290 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baa7543f-d134-4763-b418-0173c13c23ee admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:13.159086 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1248/2487] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:58:13 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 11 00:58:13.164161 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b9f7211-2aa7-4766-a7b1-609176484a05 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:13.177351 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b9f7211-2aa7-4766-a7b1-609176484a05 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:13.185322 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b042f49e-1ba6-4153-a6cf-d8a0917c42f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:13.186093 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b042f49e-1ba6-4153-a6cf-d8a0917c42f7 admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:13.186432 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b042f49e-1ba6-4153-a6cf-d8a0917c42f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:13.186871 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b042f49e-1ba6-4153-a6cf-d8a0917c42f7 admin admin] PATH_INFO: `/v3/projects/9737fe7f96574ec5823ee6129e58e4e8/users/e71635b01bc249419b4d56300994d983/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:13.204538 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b042f49e-1ba6-4153-a6cf-d8a0917c42f7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9737fe7f96574ec5823ee6129e58e4e8, user_id=e71635b01bc249419b4d56300994d983, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:13.208554 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b042f49e-1ba6-4153-a6cf-d8a0917c42f7 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:13.227925 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1240/2488] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:58:13 2022] PUT /identity/v3/projects/9737fe7f96574ec5823ee6129e58e4e8/users/e71635b01bc249419b4d56300994d983/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:58:13.232142 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa7543f-d134-4763-b418-0173c13c23ee admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:13.244792 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa7543f-d134-4763-b418-0173c13c23ee admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:13.313003 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323ebb07-6c83-4913-8606-9d3ef4d83ae6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:13.313926 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323ebb07-6c83-4913-8606-9d3ef4d83ae6 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:13.314209 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323ebb07-6c83-4913-8606-9d3ef4d83ae6 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:13.314479 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323ebb07-6c83-4913-8606-9d3ef4d83ae6 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:13.315179 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-323ebb07-6c83-4913-8606-9d3ef4d83ae6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:13.316908 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-323ebb07-6c83-4913-8606-9d3ef4d83ae6 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:13.328770 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1249/2489] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:58:13 2022] GET /identity/v3/roles => generated 1420 bytes in 97 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:58:13.333284 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b042f49e-1ba6-4153-a6cf-d8a0917c42f7 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:13.347145 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b042f49e-1ba6-4153-a6cf-d8a0917c42f7 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:13.352559 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7aa59a-0034-4cca-8514-1b5ade52c784 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:13.353302 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7aa59a-0034-4cca-8514-1b5ade52c784 admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:13.353641 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7aa59a-0034-4cca-8514-1b5ade52c784 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:13.353976 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7aa59a-0034-4cca-8514-1b5ade52c784 admin admin] PATH_INFO: `/v3/projects/9737fe7f96574ec5823ee6129e58e4e8/users/e71635b01bc249419b4d56300994d983/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:13.356006 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d7aa59a-0034-4cca-8514-1b5ade52c784 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9737fe7f96574ec5823ee6129e58e4e8, user_id=e71635b01bc249419b4d56300994d983, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:13.358767 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d7aa59a-0034-4cca-8514-1b5ade52c784 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:13.370608 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1241/2490] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:58:13 2022] PUT /identity/v3/projects/9737fe7f96574ec5823ee6129e58e4e8/users/e71635b01bc249419b4d56300994d983/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:58:13.380503 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4d24fb-b8c3-4983-a883-9088fde5a7e9 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:13.380786 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4d24fb-b8c3-4983-a883-9088fde5a7e9 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:13.381054 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4d24fb-b8c3-4983-a883-9088fde5a7e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:13.410875 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-3b4d24fb-b8c3-4983-a883-9088fde5a7e9 None None] MFA Rules not processed for user `e71635b01bc249419b4d56300994d983`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:58:13.412248 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-3b4d24fb-b8c3-4983-a883-9088fde5a7e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:13.496861 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1250/2491] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:58:13 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:58:13.512077 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7aa59a-0034-4cca-8514-1b5ade52c784 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:13.514132 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-0d7aa59a-0034-4cca-8514-1b5ade52c784 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:13.601989 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7aa59a-0034-4cca-8514-1b5ade52c784 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:13.606825 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21e737f0-9b81-41ac-8e56-d8b8e202db7a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:13.608044 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e737f0-9b81-41ac-8e56-d8b8e202db7a service cinder] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:13.608044 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e737f0-9b81-41ac-8e56-d8b8e202db7a service cinder] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:13.608044 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e737f0-9b81-41ac-8e56-d8b8e202db7a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:13.621957 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21e737f0-9b81-41ac-8e56-d8b8e202db7a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:13.624116 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21e737f0-9b81-41ac-8e56-d8b8e202db7a service cinder] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:13.651334 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1242/2492] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:58:13 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:58:14.024427 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4d24fb-b8c3-4983-a883-9088fde5a7e9 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:14.034586 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4d24fb-b8c3-4983-a883-9088fde5a7e9 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:14.040048 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-357a5a78-3d73-413c-8c25-a2f0c76df501 tempest-VolumeQuotaClassesTest-2069263752 tempest-VolumeQuotaClassesTest-2069263752-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e71635b01bc249419b4d56300994d983', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9737fe7f96574ec5823ee6129e58e4e8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:14.040655 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-357a5a78-3d73-413c-8c25-a2f0c76df501 tempest-VolumeQuotaClassesTest-2069263752 tempest-VolumeQuotaClassesTest-2069263752-project-admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:14.040948 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-357a5a78-3d73-413c-8c25-a2f0c76df501 tempest-VolumeQuotaClassesTest-2069263752 tempest-VolumeQuotaClassesTest-2069263752-project-admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:14.041209 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-357a5a78-3d73-413c-8c25-a2f0c76df501 tempest-VolumeQuotaClassesTest-2069263752 tempest-VolumeQuotaClassesTest-2069263752-project-admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:14.041855 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-357a5a78-3d73-413c-8c25-a2f0c76df501 tempest-VolumeQuotaClassesTest-2069263752 tempest-VolumeQuotaClassesTest-2069263752-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:58:14.042157 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-357a5a78-3d73-413c-8c25-a2f0c76df501 tempest-VolumeQuotaClassesTest-2069263752 tempest-VolumeQuotaClassesTest-2069263752-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:14.043806 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-357a5a78-3d73-413c-8c25-a2f0c76df501 tempest-VolumeQuotaClassesTest-2069263752 tempest-VolumeQuotaClassesTest-2069263752-project-admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:14.056148 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1251/2493] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 00:58:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:58:14.060257 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21e737f0-9b81-41ac-8e56-d8b8e202db7a service cinder] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:14.070152 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21e737f0-9b81-41ac-8e56-d8b8e202db7a service cinder] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:14.075270 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec13657-3c64-402f-a2f1-6750fed00f8e tempest-VolumeQuotaClassesTest-2069263752 tempest-VolumeQuotaClassesTest-2069263752-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e71635b01bc249419b4d56300994d983', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9737fe7f96574ec5823ee6129e58e4e8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:14.076245 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec13657-3c64-402f-a2f1-6750fed00f8e tempest-VolumeQuotaClassesTest-2069263752 tempest-VolumeQuotaClassesTest-2069263752-project-admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:14.076547 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec13657-3c64-402f-a2f1-6750fed00f8e tempest-VolumeQuotaClassesTest-2069263752 tempest-VolumeQuotaClassesTest-2069263752-project-admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:14.076814 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec13657-3c64-402f-a2f1-6750fed00f8e tempest-VolumeQuotaClassesTest-2069263752 tempest-VolumeQuotaClassesTest-2069263752-project-admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:14.077458 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ec13657-3c64-402f-a2f1-6750fed00f8e tempest-VolumeQuotaClassesTest-2069263752 tempest-VolumeQuotaClassesTest-2069263752-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:14.079397 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ec13657-3c64-402f-a2f1-6750fed00f8e tempest-VolumeQuotaClassesTest-2069263752 tempest-VolumeQuotaClassesTest-2069263752-project-admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:14.109685 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1243/2494] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:58:14 2022] POST /identity/v3/projects => generated 352 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:58:14.113688 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-357a5a78-3d73-413c-8c25-a2f0c76df501 tempest-VolumeQuotaClassesTest-2069263752 tempest-VolumeQuotaClassesTest-2069263752-project-admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:14.124640 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-357a5a78-3d73-413c-8c25-a2f0c76df501 tempest-VolumeQuotaClassesTest-2069263752 tempest-VolumeQuotaClassesTest-2069263752-project-admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:14.175608 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=100 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:58:14.184227 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d25c5a1f-3beb-4090-a5f6-5ce778c523ab tempest-VolumeQuotaClassesTest-2069263752 tempest-VolumeQuotaClassesTest-2069263752-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e71635b01bc249419b4d56300994d983', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9737fe7f96574ec5823ee6129e58e4e8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:14.184848 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25c5a1f-3beb-4090-a5f6-5ce778c523ab tempest-VolumeQuotaClassesTest-2069263752 tempest-VolumeQuotaClassesTest-2069263752-project-admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:14.185128 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25c5a1f-3beb-4090-a5f6-5ce778c523ab tempest-VolumeQuotaClassesTest-2069263752 tempest-VolumeQuotaClassesTest-2069263752-project-admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:14.185547 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25c5a1f-3beb-4090-a5f6-5ce778c523ab tempest-VolumeQuotaClassesTest-2069263752 tempest-VolumeQuotaClassesTest-2069263752-project-admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:14.186207 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d25c5a1f-3beb-4090-a5f6-5ce778c523ab tempest-VolumeQuotaClassesTest-2069263752 tempest-VolumeQuotaClassesTest-2069263752-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:58:14.186533 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d25c5a1f-3beb-4090-a5f6-5ce778c523ab tempest-VolumeQuotaClassesTest-2069263752 tempest-VolumeQuotaClassesTest-2069263752-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:14.188202 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d25c5a1f-3beb-4090-a5f6-5ce778c523ab tempest-VolumeQuotaClassesTest-2069263752 tempest-VolumeQuotaClassesTest-2069263752-project-admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:14.208342 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1252/2495] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 00:58:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:58:14.268172 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec13657-3c64-402f-a2f1-6750fed00f8e tempest-VolumeQuotaClassesTest-2069263752 tempest-VolumeQuotaClassesTest-2069263752-project-admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:14.286740 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec13657-3c64-402f-a2f1-6750fed00f8e tempest-VolumeQuotaClassesTest-2069263752 tempest-VolumeQuotaClassesTest-2069263752-project-admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:14.295209 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4653c341-6123-4bd6-9182-187eafb567cb tempest-VolumeQuotaClassesTest-2069263752 tempest-VolumeQuotaClassesTest-2069263752-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e71635b01bc249419b4d56300994d983', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9737fe7f96574ec5823ee6129e58e4e8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:14.296094 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4653c341-6123-4bd6-9182-187eafb567cb tempest-VolumeQuotaClassesTest-2069263752 tempest-VolumeQuotaClassesTest-2069263752-project-admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:14.296383 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4653c341-6123-4bd6-9182-187eafb567cb tempest-VolumeQuotaClassesTest-2069263752 tempest-VolumeQuotaClassesTest-2069263752-project-admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:14.296660 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4653c341-6123-4bd6-9182-187eafb567cb tempest-VolumeQuotaClassesTest-2069263752 tempest-VolumeQuotaClassesTest-2069263752-project-admin] PATH_INFO: `/v3/projects/67c528edc2dc4fef97c078e194a60851` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:14.298146 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4653c341-6123-4bd6-9182-187eafb567cb tempest-VolumeQuotaClassesTest-2069263752 tempest-VolumeQuotaClassesTest-2069263752-project-admin] RBAC: Authorizing `identity:delete_project(project_id=67c528edc2dc4fef97c078e194a60851)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:14.300492 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4653c341-6123-4bd6-9182-187eafb567cb tempest-VolumeQuotaClassesTest-2069263752 tempest-VolumeQuotaClassesTest-2069263752-project-admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:14.358031 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=91,keystone:INSERT=6,keystone:DELETE=3 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:58:14.365808 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-4653c341-6123-4bd6-9182-187eafb567cb tempest-VolumeQuotaClassesTest-2069263752 tempest-VolumeQuotaClassesTest-2069263752-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '67c528edc2dc4fef97c078e194a60851', 'request_id': 'req-4653c341-6123-4bd6-9182-187eafb567cb'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:14.374448 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-4653c341-6123-4bd6-9182-187eafb567cb tempest-VolumeQuotaClassesTest-2069263752 tempest-VolumeQuotaClassesTest-2069263752-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '67c528edc2dc4fef97c078e194a60851', 'request_id': 'req-4653c341-6123-4bd6-9182-187eafb567cb'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:14.380785 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-4653c341-6123-4bd6-9182-187eafb567cb tempest-VolumeQuotaClassesTest-2069263752 tempest-VolumeQuotaClassesTest-2069263752-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '67c528edc2dc4fef97c078e194a60851', 'request_id': 'req-4653c341-6123-4bd6-9182-187eafb567cb'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:14.395280 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-4653c341-6123-4bd6-9182-187eafb567cb tempest-VolumeQuotaClassesTest-2069263752 tempest-VolumeQuotaClassesTest-2069263752-project-admin] The token cache is being invalidate because project 67c528edc2dc4fef97c078e194a60851 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87923) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:58:14.395575 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-4653c341-6123-4bd6-9182-187eafb567cb tempest-VolumeQuotaClassesTest-2069263752 tempest-VolumeQuotaClassesTest-2069263752-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:14.396244 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-4653c341-6123-4bd6-9182-187eafb567cb tempest-VolumeQuotaClassesTest-2069263752 tempest-VolumeQuotaClassesTest-2069263752-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:14.397789 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1244/2496] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:58:14 2022] DELETE /identity/v3/projects/67c528edc2dc4fef97c078e194a60851 => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:58:14.597124 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d25c5a1f-3beb-4090-a5f6-5ce778c523ab tempest-VolumeQuotaClassesTest-2069263752 tempest-VolumeQuotaClassesTest-2069263752-project-admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:14.599119 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-d25c5a1f-3beb-4090-a5f6-5ce778c523ab tempest-VolumeQuotaClassesTest-2069263752 tempest-VolumeQuotaClassesTest-2069263752-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:14.699569 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d25c5a1f-3beb-4090-a5f6-5ce778c523ab tempest-VolumeQuotaClassesTest-2069263752 tempest-VolumeQuotaClassesTest-2069263752-project-admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:14.784874 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d955a97-8fd1-44bf-a3cc-0a30d817db18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:14.785526 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d955a97-8fd1-44bf-a3cc-0a30d817db18 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:14.785805 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d955a97-8fd1-44bf-a3cc-0a30d817db18 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:14.786076 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d955a97-8fd1-44bf-a3cc-0a30d817db18 admin admin] PATH_INFO: `/v3/users/50a714abb86f42a19c6dc860a489d6d3` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:14.787273 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d955a97-8fd1-44bf-a3cc-0a30d817db18 admin admin] RBAC: Authorizing `identity:delete_user(user_id=50a714abb86f42a19c6dc860a489d6d3)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:14.789853 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d955a97-8fd1-44bf-a3cc-0a30d817db18 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:14.868156 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-3d955a97-8fd1-44bf-a3cc-0a30d817db18 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '50a714abb86f42a19c6dc860a489d6d3'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:14.870202 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-3d955a97-8fd1-44bf-a3cc-0a30d817db18 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '50a714abb86f42a19c6dc860a489d6d3'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:14.872094 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-3d955a97-8fd1-44bf-a3cc-0a30d817db18 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '50a714abb86f42a19c6dc860a489d6d3'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:14.882642 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-3d955a97-8fd1-44bf-a3cc-0a30d817db18 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '50a714abb86f42a19c6dc860a489d6d3'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:14.898928 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-3d955a97-8fd1-44bf-a3cc-0a30d817db18 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '50a714abb86f42a19c6dc860a489d6d3'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:14.917564 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1253/2497] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:58:14 2022] DELETE /identity/v3/users/50a714abb86f42a19c6dc860a489d6d3 => generated 0 bytes in 321 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:58:14.922132 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc55a2d-41ee-4f35-83dd-6add8eb18d54 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:14.922411 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc55a2d-41ee-4f35-83dd-6add8eb18d54 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:14.922678 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc55a2d-41ee-4f35-83dd-6add8eb18d54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:14.950707 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-6fc55a2d-41ee-4f35-83dd-6add8eb18d54 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:58:14.951821 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-6fc55a2d-41ee-4f35-83dd-6add8eb18d54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:15.046110 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1245/2498] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:58:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:58:15.056597 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d955a97-8fd1-44bf-a3cc-0a30d817db18 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:15.058239 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-3d955a97-8fd1-44bf-a3cc-0a30d817db18 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:15.079441 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:58:15.136106 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d955a97-8fd1-44bf-a3cc-0a30d817db18 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:15.136850 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5249d66-f73f-4b3b-b6c9-0a5c8ec77746 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:15.141698 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5249d66-f73f-4b3b-b6c9-0a5c8ec77746 service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:15.141698 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5249d66-f73f-4b3b-b6c9-0a5c8ec77746 service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:15.141698 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5249d66-f73f-4b3b-b6c9-0a5c8ec77746 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:15.151196 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5249d66-f73f-4b3b-b6c9-0a5c8ec77746 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:15.152664 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5249d66-f73f-4b3b-b6c9-0a5c8ec77746 service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:15.162691 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1254/2499] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:58:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:58:15.485669 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc55a2d-41ee-4f35-83dd-6add8eb18d54 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:15.487193 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-6fc55a2d-41ee-4f35-83dd-6add8eb18d54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:15.500956 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc55a2d-41ee-4f35-83dd-6add8eb18d54 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:15.506550 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb6af278-4d38-49e2-bb3d-ef6e42231ad7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:15.507256 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb6af278-4d38-49e2-bb3d-ef6e42231ad7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:15.507781 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb6af278-4d38-49e2-bb3d-ef6e42231ad7 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:15.508061 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb6af278-4d38-49e2-bb3d-ef6e42231ad7 admin admin] PATH_INFO: `/v3/projects/655a706eb2644a7abb0740a38248f6ac` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:15.509202 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb6af278-4d38-49e2-bb3d-ef6e42231ad7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=655a706eb2644a7abb0740a38248f6ac)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:15.511307 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb6af278-4d38-49e2-bb3d-ef6e42231ad7 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:15.560189 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-bb6af278-4d38-49e2-bb3d-ef6e42231ad7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '655a706eb2644a7abb0740a38248f6ac', 'request_id': 'req-bb6af278-4d38-49e2-bb3d-ef6e42231ad7'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:15.566842 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-bb6af278-4d38-49e2-bb3d-ef6e42231ad7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '655a706eb2644a7abb0740a38248f6ac', 'request_id': 'req-bb6af278-4d38-49e2-bb3d-ef6e42231ad7'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:15.572756 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-bb6af278-4d38-49e2-bb3d-ef6e42231ad7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '655a706eb2644a7abb0740a38248f6ac', 'request_id': 'req-bb6af278-4d38-49e2-bb3d-ef6e42231ad7'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:15.586912 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-bb6af278-4d38-49e2-bb3d-ef6e42231ad7 admin admin] The token cache is being invalidate because project 655a706eb2644a7abb0740a38248f6ac was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87923) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:58:15.587197 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-bb6af278-4d38-49e2-bb3d-ef6e42231ad7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:15.587838 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-bb6af278-4d38-49e2-bb3d-ef6e42231ad7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:15.589312 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1246/2500] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:58:15 2022] DELETE /identity/v3/projects/655a706eb2644a7abb0740a38248f6ac => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:58:15.593350 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5249d66-f73f-4b3b-b6c9-0a5c8ec77746 service neutron] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:15.603889 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5249d66-f73f-4b3b-b6c9-0a5c8ec77746 service neutron] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:15.660858 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-576e70d9-2045-46fd-9450-3ae621f544af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:15.661502 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576e70d9-2045-46fd-9450-3ae621f544af admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:15.661783 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576e70d9-2045-46fd-9450-3ae621f544af admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:15.662055 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576e70d9-2045-46fd-9450-3ae621f544af admin admin] PATH_INFO: `/v3/users/e71635b01bc249419b4d56300994d983` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:15.663846 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-576e70d9-2045-46fd-9450-3ae621f544af admin admin] RBAC: Authorizing `identity:delete_user(user_id=e71635b01bc249419b4d56300994d983)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:15.665761 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-576e70d9-2045-46fd-9450-3ae621f544af admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:15.713954 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-576e70d9-2045-46fd-9450-3ae621f544af admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e71635b01bc249419b4d56300994d983'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:15.714599 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-576e70d9-2045-46fd-9450-3ae621f544af admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e71635b01bc249419b4d56300994d983'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:15.715283 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-576e70d9-2045-46fd-9450-3ae621f544af admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e71635b01bc249419b4d56300994d983'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:15.722327 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-576e70d9-2045-46fd-9450-3ae621f544af admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e71635b01bc249419b4d56300994d983'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:15.736295 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-576e70d9-2045-46fd-9450-3ae621f544af admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e71635b01bc249419b4d56300994d983'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:15.750758 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1255/2501] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:58:15 2022] DELETE /identity/v3/users/e71635b01bc249419b4d56300994d983 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:58:16.027769 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb6af278-4d38-49e2-bb3d-ef6e42231ad7 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:16.028951 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-bb6af278-4d38-49e2-bb3d-ef6e42231ad7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:16.042951 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb6af278-4d38-49e2-bb3d-ef6e42231ad7 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:16.099285 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=87,keystone:DELETE=11,keystone:INSERT=2 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:58:16.111871 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ffb3e0-5f3f-416a-aeca-92630ef5c860 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:16.112596 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ffb3e0-5f3f-416a-aeca-92630ef5c860 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:16.112938 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ffb3e0-5f3f-416a-aeca-92630ef5c860 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:16.113275 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ffb3e0-5f3f-416a-aeca-92630ef5c860 admin admin] PATH_INFO: `/v3/projects/9737fe7f96574ec5823ee6129e58e4e8` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:16.114473 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94ffb3e0-5f3f-416a-aeca-92630ef5c860 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9737fe7f96574ec5823ee6129e58e4e8)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:16.116491 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94ffb3e0-5f3f-416a-aeca-92630ef5c860 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:16.184757 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-94ffb3e0-5f3f-416a-aeca-92630ef5c860 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9737fe7f96574ec5823ee6129e58e4e8', 'request_id': 'req-94ffb3e0-5f3f-416a-aeca-92630ef5c860'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:16.191981 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-94ffb3e0-5f3f-416a-aeca-92630ef5c860 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9737fe7f96574ec5823ee6129e58e4e8', 'request_id': 'req-94ffb3e0-5f3f-416a-aeca-92630ef5c860'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:16.199673 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-94ffb3e0-5f3f-416a-aeca-92630ef5c860 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9737fe7f96574ec5823ee6129e58e4e8', 'request_id': 'req-94ffb3e0-5f3f-416a-aeca-92630ef5c860'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:16.215829 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-94ffb3e0-5f3f-416a-aeca-92630ef5c860 admin admin] The token cache is being invalidate because project 9737fe7f96574ec5823ee6129e58e4e8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87923) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:58:16.216138 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-94ffb3e0-5f3f-416a-aeca-92630ef5c860 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:16.216912 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-94ffb3e0-5f3f-416a-aeca-92630ef5c860 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:16.218975 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1247/2502] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:58:16 2022] DELETE /identity/v3/projects/9737fe7f96574ec5823ee6129e58e4e8 => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:58:16.227951 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a20e4e-2e34-4ddb-b612-95926b0f53df None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:16.228429 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a20e4e-2e34-4ddb-b612-95926b0f53df None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:16.228534 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a20e4e-2e34-4ddb-b612-95926b0f53df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:16.249409 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-29a20e4e-2e34-4ddb-b612-95926b0f53df None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:58:16.250714 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-29a20e4e-2e34-4ddb-b612-95926b0f53df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:16.300556 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:58:16.351156 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1256/2503] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:58:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:58:16.351712 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9937d955-6631-430d-9a69-6c885da80a58 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:16.352367 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9937d955-6631-430d-9a69-6c885da80a58 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:16.352595 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9937d955-6631-430d-9a69-6c885da80a58 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:16.384286 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-9937d955-6631-430d-9a69-6c885da80a58 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:58:16.385509 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-9937d955-6631-430d-9a69-6c885da80a58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:16.394839 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1248/2504] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:58:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:58:16.398932 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29a20e4e-2e34-4ddb-b612-95926b0f53df None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:16.410186 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29a20e4e-2e34-4ddb-b612-95926b0f53df None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:16.417170 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e17e2b71-f09f-49f1-8436-1944c66ead32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:16.417170 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17e2b71-f09f-49f1-8436-1944c66ead32 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:16.417170 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17e2b71-f09f-49f1-8436-1944c66ead32 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:16.417532 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17e2b71-f09f-49f1-8436-1944c66ead32 admin admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:16.418078 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e17e2b71-f09f-49f1-8436-1944c66ead32 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:58:16.418443 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e17e2b71-f09f-49f1-8436-1944c66ead32 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:16.420248 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e17e2b71-f09f-49f1-8436-1944c66ead32 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:16.434421 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1257/2505] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 00:58:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:58:16.439898 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9937d955-6631-430d-9a69-6c885da80a58 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:16.450528 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9937d955-6631-430d-9a69-6c885da80a58 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:16.456725 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dbdc409-a433-4bad-95dd-2f2d0c75d84f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:16.457341 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dbdc409-a433-4bad-95dd-2f2d0c75d84f admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:16.457613 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dbdc409-a433-4bad-95dd-2f2d0c75d84f admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:16.457877 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dbdc409-a433-4bad-95dd-2f2d0c75d84f admin admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:16.458515 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dbdc409-a433-4bad-95dd-2f2d0c75d84f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:16.460607 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dbdc409-a433-4bad-95dd-2f2d0c75d84f admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:16.490865 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1249/2506] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:58:16 2022] POST /identity/v3/projects => generated 383 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:58:16.495338 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e17e2b71-f09f-49f1-8436-1944c66ead32 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:16.505911 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e17e2b71-f09f-49f1-8436-1944c66ead32 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:16.562815 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6a8bcf6-6c71-4651-b670-0b665f26b707 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:16.563369 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a8bcf6-6c71-4651-b670-0b665f26b707 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:16.563611 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a8bcf6-6c71-4651-b670-0b665f26b707 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:16.563858 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a8bcf6-6c71-4651-b670-0b665f26b707 admin admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:16.564482 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6a8bcf6-6c71-4651-b670-0b665f26b707 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:16.566553 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6a8bcf6-6c71-4651-b670-0b665f26b707 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:16.590315 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1258/2507] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:58:16 2022] POST /identity/v3/users => generated 292 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:58:16.594602 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dbdc409-a433-4bad-95dd-2f2d0c75d84f admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:16.605495 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dbdc409-a433-4bad-95dd-2f2d0c75d84f admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:16.610183 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0855e999-59ba-4649-a1ce-6f2e15d49236 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:16.610891 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0855e999-59ba-4649-a1ce-6f2e15d49236 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:16.611230 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0855e999-59ba-4649-a1ce-6f2e15d49236 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:16.611592 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0855e999-59ba-4649-a1ce-6f2e15d49236 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:16.612333 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0855e999-59ba-4649-a1ce-6f2e15d49236 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:16.613975 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0855e999-59ba-4649-a1ce-6f2e15d49236 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:16.625995 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1250/2508] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:58:16 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:58:16.630039 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6a8bcf6-6c71-4651-b670-0b665f26b707 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:16.640898 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6a8bcf6-6c71-4651-b670-0b665f26b707 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:16.645737 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89a3a2d5-1145-46b4-bb3a-d15acf00e146 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:16.646334 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a3a2d5-1145-46b4-bb3a-d15acf00e146 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:16.646611 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a3a2d5-1145-46b4-bb3a-d15acf00e146 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:16.646913 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a3a2d5-1145-46b4-bb3a-d15acf00e146 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:16.647615 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89a3a2d5-1145-46b4-bb3a-d15acf00e146 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:16.649558 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89a3a2d5-1145-46b4-bb3a-d15acf00e146 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:16.660961 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1259/2509] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:58:16 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:58:16.665315 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0855e999-59ba-4649-a1ce-6f2e15d49236 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:16.675465 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0855e999-59ba-4649-a1ce-6f2e15d49236 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:16.680938 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc73ab73-0b14-4340-9243-de1fe5b357d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:16.681593 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc73ab73-0b14-4340-9243-de1fe5b357d7 admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:16.681874 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc73ab73-0b14-4340-9243-de1fe5b357d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:16.682150 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc73ab73-0b14-4340-9243-de1fe5b357d7 admin admin] PATH_INFO: `/v3/projects/8e4780f7263a40f89a08f5616cfc6ae3/users/49549047248b4072a7cd4d1e44a9b9a9/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:16.699210 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc73ab73-0b14-4340-9243-de1fe5b357d7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8e4780f7263a40f89a08f5616cfc6ae3, user_id=49549047248b4072a7cd4d1e44a9b9a9, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:16.701907 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc73ab73-0b14-4340-9243-de1fe5b357d7 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:16.715129 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1251/2510] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:58:16 2022] PUT /identity/v3/projects/8e4780f7263a40f89a08f5616cfc6ae3/users/49549047248b4072a7cd4d1e44a9b9a9/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:58:16.726185 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e969ec9e-24ff-4ee7-b835-dabef875ceeb None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:16.726546 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e969ec9e-24ff-4ee7-b835-dabef875ceeb None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:16.726929 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e969ec9e-24ff-4ee7-b835-dabef875ceeb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:16.755377 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-e969ec9e-24ff-4ee7-b835-dabef875ceeb None None] MFA Rules not processed for user `49549047248b4072a7cd4d1e44a9b9a9`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:58:16.756586 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-e969ec9e-24ff-4ee7-b835-dabef875ceeb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:16.830882 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1260/2511] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:58:16 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:58:16.833948 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc73ab73-0b14-4340-9243-de1fe5b357d7 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:16.905487 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc73ab73-0b14-4340-9243-de1fe5b357d7 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:16.911052 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b15e08-c0d7-407e-887e-2c91cd13c17b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:16.911663 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b15e08-c0d7-407e-887e-2c91cd13c17b admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:16.912070 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b15e08-c0d7-407e-887e-2c91cd13c17b admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:16.912348 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b15e08-c0d7-407e-887e-2c91cd13c17b admin admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:16.913874 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9b15e08-c0d7-407e-887e-2c91cd13c17b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:16.915568 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9b15e08-c0d7-407e-887e-2c91cd13c17b admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:16.946646 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1252/2512] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:58:16 2022] POST /identity/v3/projects => generated 385 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:58:16.950561 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e969ec9e-24ff-4ee7-b835-dabef875ceeb None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:16.962488 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e969ec9e-24ff-4ee7-b835-dabef875ceeb None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:16.998701 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:58:17.037277 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-418697c3-9d0e-4264-9b33-81f3e28591fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:17.038618 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418697c3-9d0e-4264-9b33-81f3e28591fb admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:17.038803 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418697c3-9d0e-4264-9b33-81f3e28591fb admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:17.039339 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418697c3-9d0e-4264-9b33-81f3e28591fb admin admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:17.039847 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-418697c3-9d0e-4264-9b33-81f3e28591fb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:17.043324 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-418697c3-9d0e-4264-9b33-81f3e28591fb admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:17.076650 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1261/2513] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:58:16 2022] POST /identity/v3/users => generated 299 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:58:17.078088 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b15e08-c0d7-407e-887e-2c91cd13c17b admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:17.089530 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:58:17.094283 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b15e08-c0d7-407e-887e-2c91cd13c17b admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:17.111810 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-721696f6-958c-43a1-b625-767b285d72fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:17.113282 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721696f6-958c-43a1-b625-767b285d72fc admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:17.113282 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721696f6-958c-43a1-b625-767b285d72fc admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:17.113509 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721696f6-958c-43a1-b625-767b285d72fc admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:17.114317 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-721696f6-958c-43a1-b625-767b285d72fc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:17.116788 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-721696f6-958c-43a1-b625-767b285d72fc admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:17.128462 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1253/2514] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:58:17 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:58:17.133525 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-418697c3-9d0e-4264-9b33-81f3e28591fb admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:17.144601 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-418697c3-9d0e-4264-9b33-81f3e28591fb admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:17.150702 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72700909-287a-4739-988f-658f6ddab141 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:17.151561 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72700909-287a-4739-988f-658f6ddab141 admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:17.152372 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72700909-287a-4739-988f-658f6ddab141 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:17.152372 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72700909-287a-4739-988f-658f6ddab141 admin admin] PATH_INFO: `/v3/projects/d1c10d7e13f441a2b55440e942d3ec15/users/ef1480a4420a4f4794055f17c533b48d/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:17.171533 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72700909-287a-4739-988f-658f6ddab141 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d1c10d7e13f441a2b55440e942d3ec15, user_id=ef1480a4420a4f4794055f17c533b48d, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:17.174219 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72700909-287a-4739-988f-658f6ddab141 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:17.193086 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1262/2515] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:58:17 2022] PUT /identity/v3/projects/d1c10d7e13f441a2b55440e942d3ec15/users/ef1480a4420a4f4794055f17c533b48d/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:58:17.195293 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-721696f6-958c-43a1-b625-767b285d72fc admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:17.206985 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-721696f6-958c-43a1-b625-767b285d72fc admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:17.268761 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-448ad5b7-60ce-4300-8c0d-b8cd768d368e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:17.269479 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-448ad5b7-60ce-4300-8c0d-b8cd768d368e admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:17.269824 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-448ad5b7-60ce-4300-8c0d-b8cd768d368e admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:17.270169 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-448ad5b7-60ce-4300-8c0d-b8cd768d368e admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:17.270942 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-448ad5b7-60ce-4300-8c0d-b8cd768d368e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:17.273167 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-448ad5b7-60ce-4300-8c0d-b8cd768d368e admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:17.287577 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1254/2516] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:58:17 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:58:17.289510 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72700909-287a-4739-988f-658f6ddab141 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:17.301379 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72700909-287a-4739-988f-658f6ddab141 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:17.307648 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e888204-30d5-46a1-bd9f-85e062b33944 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:17.308393 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e888204-30d5-46a1-bd9f-85e062b33944 admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:17.308742 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e888204-30d5-46a1-bd9f-85e062b33944 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:17.309059 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e888204-30d5-46a1-bd9f-85e062b33944 admin admin] PATH_INFO: `/v3/projects/d1c10d7e13f441a2b55440e942d3ec15/users/ef1480a4420a4f4794055f17c533b48d/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:17.311073 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e888204-30d5-46a1-bd9f-85e062b33944 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d1c10d7e13f441a2b55440e942d3ec15, user_id=ef1480a4420a4f4794055f17c533b48d, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:17.313730 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e888204-30d5-46a1-bd9f-85e062b33944 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:17.325775 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1263/2517] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:58:17 2022] PUT /identity/v3/projects/d1c10d7e13f441a2b55440e942d3ec15/users/ef1480a4420a4f4794055f17c533b48d/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:58:17.338051 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e8e368-c9a5-4001-a5f1-778a095474e5 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:17.338561 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e8e368-c9a5-4001-a5f1-778a095474e5 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:17.338888 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e8e368-c9a5-4001-a5f1-778a095474e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:17.373365 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-c7e8e368-c9a5-4001-a5f1-778a095474e5 None None] MFA Rules not processed for user `ef1480a4420a4f4794055f17c533b48d`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:58:17.374641 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-c7e8e368-c9a5-4001-a5f1-778a095474e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:17.458281 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1255/2518] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:58:17 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:58:17.471144 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e888204-30d5-46a1-bd9f-85e062b33944 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:17.472951 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-5e888204-30d5-46a1-bd9f-85e062b33944 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:17.538978 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e888204-30d5-46a1-bd9f-85e062b33944 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:17.542642 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab45ff46-3972-4613-b2c0-e0a95d3719e8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:17.543298 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab45ff46-3972-4613-b2c0-e0a95d3719e8 service cinder] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:17.543574 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab45ff46-3972-4613-b2c0-e0a95d3719e8 service cinder] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:17.543845 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab45ff46-3972-4613-b2c0-e0a95d3719e8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:17.605279 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab45ff46-3972-4613-b2c0-e0a95d3719e8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:17.606684 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab45ff46-3972-4613-b2c0-e0a95d3719e8 service cinder] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:17.616525 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1264/2519] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:58:17 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:58:18.692750 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e8e368-c9a5-4001-a5f1-778a095474e5 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:18.694169 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-c7e8e368-c9a5-4001-a5f1-778a095474e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:18.755794 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab45ff46-3972-4613-b2c0-e0a95d3719e8 service cinder] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:18.757878 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-ab45ff46-3972-4613-b2c0-e0a95d3719e8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:18.781105 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e8e368-c9a5-4001-a5f1-778a095474e5 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:18.786426 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c71d97-aa7b-4686-954a-dfb8c4b50223 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cdbfc2ccd73e4a5eb8c999be5b094619', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:18.787058 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c71d97-aa7b-4686-954a-dfb8c4b50223 service placement] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:18.787340 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c71d97-aa7b-4686-954a-dfb8c4b50223 service placement] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:18.787610 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c71d97-aa7b-4686-954a-dfb8c4b50223 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:18.789888 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-14c71d97-aa7b-4686-954a-dfb8c4b50223 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:18.804253 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14c71d97-aa7b-4686-954a-dfb8c4b50223 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:18.805803 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14c71d97-aa7b-4686-954a-dfb8c4b50223 service placement] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:18.807484 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:58:18.816006 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1256/2520] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:58:18 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:58:18.831111 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab45ff46-3972-4613-b2c0-e0a95d3719e8 service cinder] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:18.836914 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f2f441f-93b6-43cf-beaf-488e8a0a7294 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:18.837728 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2f441f-93b6-43cf-beaf-488e8a0a7294 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:18.838130 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2f441f-93b6-43cf-beaf-488e8a0a7294 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:18.838516 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2f441f-93b6-43cf-beaf-488e8a0a7294 admin admin] PATH_INFO: `/v3/users/953d7ed2081c48c79a1a5660370b861b` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:18.839882 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f2f441f-93b6-43cf-beaf-488e8a0a7294 admin admin] RBAC: Authorizing `identity:delete_user(user_id=953d7ed2081c48c79a1a5660370b861b)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:18.842143 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f2f441f-93b6-43cf-beaf-488e8a0a7294 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:18.890248 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-5f2f441f-93b6-43cf-beaf-488e8a0a7294 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '953d7ed2081c48c79a1a5660370b861b'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:18.890846 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-5f2f441f-93b6-43cf-beaf-488e8a0a7294 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '953d7ed2081c48c79a1a5660370b861b'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:18.891451 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-5f2f441f-93b6-43cf-beaf-488e8a0a7294 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '953d7ed2081c48c79a1a5660370b861b'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:18.899198 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-5f2f441f-93b6-43cf-beaf-488e8a0a7294 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '953d7ed2081c48c79a1a5660370b861b'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:18.912038 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-5f2f441f-93b6-43cf-beaf-488e8a0a7294 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '953d7ed2081c48c79a1a5660370b861b'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:18.926385 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1265/2521] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:58:18 2022] DELETE /identity/v3/users/953d7ed2081c48c79a1a5660370b861b => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:58:19.136372 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c71d97-aa7b-4686-954a-dfb8c4b50223 service placement] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:19.143079 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=100 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:58:19.148686 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c71d97-aa7b-4686-954a-dfb8c4b50223 service placement] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:19.207504 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a88beefe-c9dd-4312-ac3b-83e6f3ebcf5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:19.208126 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88beefe-c9dd-4312-ac3b-83e6f3ebcf5f admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:19.208399 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88beefe-c9dd-4312-ac3b-83e6f3ebcf5f admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:19.208664 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88beefe-c9dd-4312-ac3b-83e6f3ebcf5f admin admin] PATH_INFO: `/v3/projects/299fcf2ee8b24a4fbb27ed83954c8e24` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:19.210666 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a88beefe-c9dd-4312-ac3b-83e6f3ebcf5f admin admin] RBAC: Authorizing `identity:delete_project(project_id=299fcf2ee8b24a4fbb27ed83954c8e24)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:19.212898 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a88beefe-c9dd-4312-ac3b-83e6f3ebcf5f admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:19.260375 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-a88beefe-c9dd-4312-ac3b-83e6f3ebcf5f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '299fcf2ee8b24a4fbb27ed83954c8e24', 'request_id': 'req-a88beefe-c9dd-4312-ac3b-83e6f3ebcf5f'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:19.267382 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-a88beefe-c9dd-4312-ac3b-83e6f3ebcf5f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '299fcf2ee8b24a4fbb27ed83954c8e24', 'request_id': 'req-a88beefe-c9dd-4312-ac3b-83e6f3ebcf5f'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:19.273072 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-a88beefe-c9dd-4312-ac3b-83e6f3ebcf5f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '299fcf2ee8b24a4fbb27ed83954c8e24', 'request_id': 'req-a88beefe-c9dd-4312-ac3b-83e6f3ebcf5f'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:19.287857 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-a88beefe-c9dd-4312-ac3b-83e6f3ebcf5f admin admin] The token cache is being invalidate because project 299fcf2ee8b24a4fbb27ed83954c8e24 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87923) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:58:19.288852 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-a88beefe-c9dd-4312-ac3b-83e6f3ebcf5f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:19.289811 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-a88beefe-c9dd-4312-ac3b-83e6f3ebcf5f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:19.291385 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1257/2522] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:58:19 2022] DELETE /identity/v3/projects/299fcf2ee8b24a4fbb27ed83954c8e24 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:58:19.295623 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f2f441f-93b6-43cf-beaf-488e8a0a7294 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:19.297066 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-5f2f441f-93b6-43cf-beaf-488e8a0a7294 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:19.311779 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f2f441f-93b6-43cf-beaf-488e8a0a7294 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:19.388505 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158c126a-dace-49ef-8554-dfbb30ef7092 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:19.388505 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158c126a-dace-49ef-8554-dfbb30ef7092 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:19.388505 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158c126a-dace-49ef-8554-dfbb30ef7092 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:19.388505 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158c126a-dace-49ef-8554-dfbb30ef7092 admin admin] PATH_INFO: `/v3/users/6e020c4fd9534e39bb2153ecdbf5b763` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:19.388880 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-158c126a-dace-49ef-8554-dfbb30ef7092 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6e020c4fd9534e39bb2153ecdbf5b763)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:19.391161 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-158c126a-dace-49ef-8554-dfbb30ef7092 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:19.450589 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-158c126a-dace-49ef-8554-dfbb30ef7092 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6e020c4fd9534e39bb2153ecdbf5b763'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:19.451268 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-158c126a-dace-49ef-8554-dfbb30ef7092 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6e020c4fd9534e39bb2153ecdbf5b763'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:19.451864 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-158c126a-dace-49ef-8554-dfbb30ef7092 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6e020c4fd9534e39bb2153ecdbf5b763'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:19.458796 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-158c126a-dace-49ef-8554-dfbb30ef7092 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6e020c4fd9534e39bb2153ecdbf5b763'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:19.464088 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=82,keystone:DELETE=17,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:58:19.472419 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-158c126a-dace-49ef-8554-dfbb30ef7092 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6e020c4fd9534e39bb2153ecdbf5b763'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:19.488079 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1266/2523] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:58:19 2022] DELETE /identity/v3/users/6e020c4fd9534e39bb2153ecdbf5b763 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:58:19.663647 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a88beefe-c9dd-4312-ac3b-83e6f3ebcf5f admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:19.665127 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-a88beefe-c9dd-4312-ac3b-83e6f3ebcf5f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:19.679308 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a88beefe-c9dd-4312-ac3b-83e6f3ebcf5f admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:19.741257 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-125cd2c7-8759-4e4c-8079-a3d20fc2c5b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:19.741971 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125cd2c7-8759-4e4c-8079-a3d20fc2c5b7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:19.742142 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125cd2c7-8759-4e4c-8079-a3d20fc2c5b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:19.742408 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125cd2c7-8759-4e4c-8079-a3d20fc2c5b7 admin admin] PATH_INFO: `/v3/projects/32390cc59f8540a7aac26e20f1b25166` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:19.743559 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-125cd2c7-8759-4e4c-8079-a3d20fc2c5b7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=32390cc59f8540a7aac26e20f1b25166)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:19.745457 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-125cd2c7-8759-4e4c-8079-a3d20fc2c5b7 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:19.783627 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:58:19.793823 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-125cd2c7-8759-4e4c-8079-a3d20fc2c5b7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '32390cc59f8540a7aac26e20f1b25166', 'request_id': 'req-125cd2c7-8759-4e4c-8079-a3d20fc2c5b7'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:19.800160 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-125cd2c7-8759-4e4c-8079-a3d20fc2c5b7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '32390cc59f8540a7aac26e20f1b25166', 'request_id': 'req-125cd2c7-8759-4e4c-8079-a3d20fc2c5b7'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:19.805804 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-125cd2c7-8759-4e4c-8079-a3d20fc2c5b7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '32390cc59f8540a7aac26e20f1b25166', 'request_id': 'req-125cd2c7-8759-4e4c-8079-a3d20fc2c5b7'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:19.817841 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-125cd2c7-8759-4e4c-8079-a3d20fc2c5b7 admin admin] The token cache is being invalidate because project 32390cc59f8540a7aac26e20f1b25166 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87923) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:58:19.818129 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-125cd2c7-8759-4e4c-8079-a3d20fc2c5b7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:19.818810 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-125cd2c7-8759-4e4c-8079-a3d20fc2c5b7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:19.821288 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1258/2524] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:58:19 2022] DELETE /identity/v3/projects/32390cc59f8540a7aac26e20f1b25166 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:58:21.951709 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158c126a-dace-49ef-8554-dfbb30ef7092 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:21.953145 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-158c126a-dace-49ef-8554-dfbb30ef7092 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:22.009723 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158c126a-dace-49ef-8554-dfbb30ef7092 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:22.058895 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a734890-aee2-48e6-9d49-55a13b3d6a52 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:22.059508 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a734890-aee2-48e6-9d49-55a13b3d6a52 service cinder] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:22.059785 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a734890-aee2-48e6-9d49-55a13b3d6a52 service cinder] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:22.060058 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a734890-aee2-48e6-9d49-55a13b3d6a52 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:22.061822 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-2a734890-aee2-48e6-9d49-55a13b3d6a52 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:22.129739 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a734890-aee2-48e6-9d49-55a13b3d6a52 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:22.131218 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a734890-aee2-48e6-9d49-55a13b3d6a52 service cinder] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:22.141946 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1267/2525] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:58:21 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 191 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:58:22.192660 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-125cd2c7-8759-4e4c-8079-a3d20fc2c5b7 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:22.195402 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-125cd2c7-8759-4e4c-8079-a3d20fc2c5b7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:22.298219 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-125cd2c7-8759-4e4c-8079-a3d20fc2c5b7 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:22.304307 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f32b29ed-c841-4f18-bb1f-e51dbd00fcce service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '835def0e6c6d4365a2767051509eb0d6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'ResellerAdmin', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:22.304920 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f32b29ed-c841-4f18-bb1f-e51dbd00fcce service nova] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:22.305228 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f32b29ed-c841-4f18-bb1f-e51dbd00fcce service nova] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:22.305503 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f32b29ed-c841-4f18-bb1f-e51dbd00fcce service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:22.307318 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-f32b29ed-c841-4f18-bb1f-e51dbd00fcce service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:22.320893 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f32b29ed-c841-4f18-bb1f-e51dbd00fcce service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:22.322574 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f32b29ed-c841-4f18-bb1f-e51dbd00fcce service nova] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:22.333515 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1259/2526] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:58:22 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:58:24.791522 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a734890-aee2-48e6-9d49-55a13b3d6a52 service cinder] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:24.792949 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-2a734890-aee2-48e6-9d49-55a13b3d6a52 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:24.808116 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a734890-aee2-48e6-9d49-55a13b3d6a52 service cinder] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:24.813533 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87ac8ce1-14d5-43fe-b63e-eaf4be521dac service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '835def0e6c6d4365a2767051509eb0d6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'ResellerAdmin', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:24.814157 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ac8ce1-14d5-43fe-b63e-eaf4be521dac service nova] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:24.814432 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ac8ce1-14d5-43fe-b63e-eaf4be521dac service nova] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:24.814698 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ac8ce1-14d5-43fe-b63e-eaf4be521dac service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:24.816379 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-87ac8ce1-14d5-43fe-b63e-eaf4be521dac service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:24.987672 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:DELETE=3,keystone:SELECT=96,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:58:25.007949 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87ac8ce1-14d5-43fe-b63e-eaf4be521dac service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:25.009622 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87ac8ce1-14d5-43fe-b63e-eaf4be521dac service nova] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:25.019935 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1268/2527] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:58:24 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 229 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:58:27.130833 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f32b29ed-c841-4f18-bb1f-e51dbd00fcce service nova] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:27.132850 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-f32b29ed-c841-4f18-bb1f-e51dbd00fcce service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:27.201890 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f32b29ed-c841-4f18-bb1f-e51dbd00fcce service nova] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:27.206396 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df3589e6-1a1a-4741-a85d-375fb12d63b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:27.207057 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3589e6-1a1a-4741-a85d-375fb12d63b5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:27.207329 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3589e6-1a1a-4741-a85d-375fb12d63b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:27.207601 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3589e6-1a1a-4741-a85d-375fb12d63b5 admin admin] PATH_INFO: `/v3/users/49549047248b4072a7cd4d1e44a9b9a9` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:27.208714 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df3589e6-1a1a-4741-a85d-375fb12d63b5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=49549047248b4072a7cd4d1e44a9b9a9)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:27.211390 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df3589e6-1a1a-4741-a85d-375fb12d63b5 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:27.259540 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-df3589e6-1a1a-4741-a85d-375fb12d63b5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '49549047248b4072a7cd4d1e44a9b9a9'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:27.260185 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-df3589e6-1a1a-4741-a85d-375fb12d63b5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '49549047248b4072a7cd4d1e44a9b9a9'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:27.260772 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-df3589e6-1a1a-4741-a85d-375fb12d63b5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '49549047248b4072a7cd4d1e44a9b9a9'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:27.265839 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:58:27.268732 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-df3589e6-1a1a-4741-a85d-375fb12d63b5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '49549047248b4072a7cd4d1e44a9b9a9'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:27.283486 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-df3589e6-1a1a-4741-a85d-375fb12d63b5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '49549047248b4072a7cd4d1e44a9b9a9'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:27.296419 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1260/2528] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:58:27 2022] DELETE /identity/v3/users/49549047248b4072a7cd4d1e44a9b9a9 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:58:27.301065 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-956605b4-bed6-4f7b-855b-9db10e27140a None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:27.301549 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-956605b4-bed6-4f7b-855b-9db10e27140a None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:27.301660 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-956605b4-bed6-4f7b-855b-9db10e27140a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:27.320962 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-956605b4-bed6-4f7b-855b-9db10e27140a None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:58:27.322143 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-956605b4-bed6-4f7b-855b-9db10e27140a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:27.379237 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1269/2529] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:58:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:58:27.389581 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df3589e6-1a1a-4741-a85d-375fb12d63b5 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:27.391002 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-df3589e6-1a1a-4741-a85d-375fb12d63b5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:27.450328 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df3589e6-1a1a-4741-a85d-375fb12d63b5 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:27.453988 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acdb071b-e372-4ad4-8993-adcb2b1b2176 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:27.454580 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acdb071b-e372-4ad4-8993-adcb2b1b2176 service neutron] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:27.454887 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acdb071b-e372-4ad4-8993-adcb2b1b2176 service neutron] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:27.455178 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acdb071b-e372-4ad4-8993-adcb2b1b2176 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:27.464637 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acdb071b-e372-4ad4-8993-adcb2b1b2176 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:27.466070 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acdb071b-e372-4ad4-8993-adcb2b1b2176 service neutron] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:27.475600 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1261/2530] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:58:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:58:27.747554 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-956605b4-bed6-4f7b-855b-9db10e27140a None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:27.749090 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-956605b4-bed6-4f7b-855b-9db10e27140a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:27.762666 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-956605b4-bed6-4f7b-855b-9db10e27140a None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:27.767646 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed235239-6545-4f01-abd0-6804725cf045 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:27.768285 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed235239-6545-4f01-abd0-6804725cf045 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:27.768822 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed235239-6545-4f01-abd0-6804725cf045 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:27.769100 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed235239-6545-4f01-abd0-6804725cf045 admin admin] PATH_INFO: `/v3/projects/8e4780f7263a40f89a08f5616cfc6ae3` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:27.770229 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed235239-6545-4f01-abd0-6804725cf045 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8e4780f7263a40f89a08f5616cfc6ae3)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:27.772383 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed235239-6545-4f01-abd0-6804725cf045 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:27.820797 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-ed235239-6545-4f01-abd0-6804725cf045 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8e4780f7263a40f89a08f5616cfc6ae3', 'request_id': 'req-ed235239-6545-4f01-abd0-6804725cf045'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:27.826861 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-ed235239-6545-4f01-abd0-6804725cf045 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8e4780f7263a40f89a08f5616cfc6ae3', 'request_id': 'req-ed235239-6545-4f01-abd0-6804725cf045'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:27.833850 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-ed235239-6545-4f01-abd0-6804725cf045 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8e4780f7263a40f89a08f5616cfc6ae3', 'request_id': 'req-ed235239-6545-4f01-abd0-6804725cf045'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:27.844303 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-ed235239-6545-4f01-abd0-6804725cf045 admin admin] The token cache is being invalidate because project 8e4780f7263a40f89a08f5616cfc6ae3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87924) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:58:27.844588 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-ed235239-6545-4f01-abd0-6804725cf045 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:27.845217 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-ed235239-6545-4f01-abd0-6804725cf045 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:27.846729 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1270/2531] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:58:27 2022] DELETE /identity/v3/projects/8e4780f7263a40f89a08f5616cfc6ae3 => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:58:27.851131 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acdb071b-e372-4ad4-8993-adcb2b1b2176 service neutron] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:27.862126 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acdb071b-e372-4ad4-8993-adcb2b1b2176 service neutron] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:27.921051 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d59068-78a6-40ea-ba0c-c3d8b0f1b864 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:27.921593 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d59068-78a6-40ea-ba0c-c3d8b0f1b864 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:27.921871 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d59068-78a6-40ea-ba0c-c3d8b0f1b864 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:27.922141 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d59068-78a6-40ea-ba0c-c3d8b0f1b864 admin admin] PATH_INFO: `/v3/users/ef1480a4420a4f4794055f17c533b48d` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:27.923282 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2d59068-78a6-40ea-ba0c-c3d8b0f1b864 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ef1480a4420a4f4794055f17c533b48d)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:27.925202 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2d59068-78a6-40ea-ba0c-c3d8b0f1b864 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:27.970026 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-a2d59068-78a6-40ea-ba0c-c3d8b0f1b864 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ef1480a4420a4f4794055f17c533b48d'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:27.970703 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-a2d59068-78a6-40ea-ba0c-c3d8b0f1b864 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ef1480a4420a4f4794055f17c533b48d'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:27.971346 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-a2d59068-78a6-40ea-ba0c-c3d8b0f1b864 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ef1480a4420a4f4794055f17c533b48d'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:27.978976 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-a2d59068-78a6-40ea-ba0c-c3d8b0f1b864 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ef1480a4420a4f4794055f17c533b48d'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:27.990645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-a2d59068-78a6-40ea-ba0c-c3d8b0f1b864 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ef1480a4420a4f4794055f17c533b48d'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:27.992830 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:58:28.005780 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1262/2532] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:58:27 2022] DELETE /identity/v3/users/ef1480a4420a4f4794055f17c533b48d => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:58:28.275697 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed235239-6545-4f01-abd0-6804725cf045 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:28.277351 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-ed235239-6545-4f01-abd0-6804725cf045 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:28.296644 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed235239-6545-4f01-abd0-6804725cf045 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:28.364982 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5472cd49-3ba3-4a94-8ab6-7be224c9f420 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:28.365753 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5472cd49-3ba3-4a94-8ab6-7be224c9f420 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:28.365926 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5472cd49-3ba3-4a94-8ab6-7be224c9f420 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:28.366194 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5472cd49-3ba3-4a94-8ab6-7be224c9f420 admin admin] PATH_INFO: `/v3/projects/d1c10d7e13f441a2b55440e942d3ec15` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:28.367348 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5472cd49-3ba3-4a94-8ab6-7be224c9f420 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d1c10d7e13f441a2b55440e942d3ec15)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:28.369268 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5472cd49-3ba3-4a94-8ab6-7be224c9f420 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:28.378968 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:58:28.420859 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-5472cd49-3ba3-4a94-8ab6-7be224c9f420 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd1c10d7e13f441a2b55440e942d3ec15', 'request_id': 'req-5472cd49-3ba3-4a94-8ab6-7be224c9f420'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:28.427382 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-5472cd49-3ba3-4a94-8ab6-7be224c9f420 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd1c10d7e13f441a2b55440e942d3ec15', 'request_id': 'req-5472cd49-3ba3-4a94-8ab6-7be224c9f420'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:28.437621 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-5472cd49-3ba3-4a94-8ab6-7be224c9f420 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd1c10d7e13f441a2b55440e942d3ec15', 'request_id': 'req-5472cd49-3ba3-4a94-8ab6-7be224c9f420'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:28.448641 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-5472cd49-3ba3-4a94-8ab6-7be224c9f420 admin admin] The token cache is being invalidate because project d1c10d7e13f441a2b55440e942d3ec15 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87924) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:58:28.448926 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-5472cd49-3ba3-4a94-8ab6-7be224c9f420 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:28.449564 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-5472cd49-3ba3-4a94-8ab6-7be224c9f420 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:28.451081 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1271/2533] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:58:28 2022] DELETE /identity/v3/projects/d1c10d7e13f441a2b55440e942d3ec15 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:58:28.457741 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694d6a7b-6a21-4d82-ae04-e4695b974aab None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:28.458222 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694d6a7b-6a21-4d82-ae04-e4695b974aab None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:28.458455 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694d6a7b-6a21-4d82-ae04-e4695b974aab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:28.479804 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-694d6a7b-6a21-4d82-ae04-e4695b974aab None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:58:28.480992 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-694d6a7b-6a21-4d82-ae04-e4695b974aab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:28.539713 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1263/2534] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:58:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:58:28.547831 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2066ce-a6be-4d92-896f-6ccf41930bc4 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:28.548119 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2066ce-a6be-4d92-896f-6ccf41930bc4 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:28.548396 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2066ce-a6be-4d92-896f-6ccf41930bc4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:28.568155 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-8f2066ce-a6be-4d92-896f-6ccf41930bc4 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:58:28.569394 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-8f2066ce-a6be-4d92-896f-6ccf41930bc4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:28.577685 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1272/2535] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:58:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:58:28.582175 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-694d6a7b-6a21-4d82-ae04-e4695b974aab None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:28.592671 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-694d6a7b-6a21-4d82-ae04-e4695b974aab None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:28.597196 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b308cd3-30fc-48fc-b9ec-e9950695f087 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:28.597785 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b308cd3-30fc-48fc-b9ec-e9950695f087 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:28.598060 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b308cd3-30fc-48fc-b9ec-e9950695f087 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:28.598327 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b308cd3-30fc-48fc-b9ec-e9950695f087 admin admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:28.599001 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b308cd3-30fc-48fc-b9ec-e9950695f087 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:58:28.599302 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b308cd3-30fc-48fc-b9ec-e9950695f087 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:28.601639 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b308cd3-30fc-48fc-b9ec-e9950695f087 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:28.612995 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1264/2536] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 00:58:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:58:28.617692 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f2066ce-a6be-4d92-896f-6ccf41930bc4 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:28.626812 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f2066ce-a6be-4d92-896f-6ccf41930bc4 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:28.631395 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b593e54-22d9-4e2c-a8f5-ff8fc9012fe6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:28.632154 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b593e54-22d9-4e2c-a8f5-ff8fc9012fe6 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:28.632429 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b593e54-22d9-4e2c-a8f5-ff8fc9012fe6 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:28.632698 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b593e54-22d9-4e2c-a8f5-ff8fc9012fe6 admin admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:28.633331 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b593e54-22d9-4e2c-a8f5-ff8fc9012fe6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:28.635232 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b593e54-22d9-4e2c-a8f5-ff8fc9012fe6 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:28.662411 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1273/2537] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:58:28 2022] POST /identity/v3/projects => generated 373 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:58:28.667295 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b308cd3-30fc-48fc-b9ec-e9950695f087 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:28.677132 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b308cd3-30fc-48fc-b9ec-e9950695f087 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:28.733282 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79172b43-e453-4ef2-bdcf-ad61c1f59a4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:28.733881 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79172b43-e453-4ef2-bdcf-ad61c1f59a4d admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:28.734160 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79172b43-e453-4ef2-bdcf-ad61c1f59a4d admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:28.734430 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79172b43-e453-4ef2-bdcf-ad61c1f59a4d admin admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:28.735093 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79172b43-e453-4ef2-bdcf-ad61c1f59a4d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:28.737003 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79172b43-e453-4ef2-bdcf-ad61c1f59a4d admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:28.758714 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1265/2538] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:58:28 2022] POST /identity/v3/users => generated 287 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:58:28.762972 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b593e54-22d9-4e2c-a8f5-ff8fc9012fe6 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:28.773847 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b593e54-22d9-4e2c-a8f5-ff8fc9012fe6 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:28.778572 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d482f09-8254-4a42-a8e5-79b3a27e556b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:28.779201 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d482f09-8254-4a42-a8e5-79b3a27e556b admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:28.779479 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d482f09-8254-4a42-a8e5-79b3a27e556b admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:28.779746 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d482f09-8254-4a42-a8e5-79b3a27e556b admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:28.780380 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d482f09-8254-4a42-a8e5-79b3a27e556b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:28.781957 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d482f09-8254-4a42-a8e5-79b3a27e556b admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:28.792125 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1274/2539] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:58:28 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:58:28.796537 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79172b43-e453-4ef2-bdcf-ad61c1f59a4d admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:28.807538 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79172b43-e453-4ef2-bdcf-ad61c1f59a4d admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:28.812307 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c06f745-c8eb-4948-9a20-34f0082d4943 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:28.812896 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c06f745-c8eb-4948-9a20-34f0082d4943 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:28.813172 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c06f745-c8eb-4948-9a20-34f0082d4943 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:28.813443 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c06f745-c8eb-4948-9a20-34f0082d4943 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:28.814082 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c06f745-c8eb-4948-9a20-34f0082d4943 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:28.815715 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c06f745-c8eb-4948-9a20-34f0082d4943 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:28.826864 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1266/2540] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:58:28 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:58:28.831119 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d482f09-8254-4a42-a8e5-79b3a27e556b admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:28.841646 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d482f09-8254-4a42-a8e5-79b3a27e556b admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:28.846653 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94263e36-44a2-4524-9209-21644dcd289a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:28.847326 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94263e36-44a2-4524-9209-21644dcd289a admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:28.847611 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94263e36-44a2-4524-9209-21644dcd289a admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:28.847888 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94263e36-44a2-4524-9209-21644dcd289a admin admin] PATH_INFO: `/v3/projects/ea41349cca9f410c942383d1b1acfada/users/78fa82aae45d447aba89cb63b8d843af/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:28.862171 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94263e36-44a2-4524-9209-21644dcd289a admin admin] RBAC: Authorizing `identity:create_grant(project_id=ea41349cca9f410c942383d1b1acfada, user_id=78fa82aae45d447aba89cb63b8d843af, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:28.864850 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94263e36-44a2-4524-9209-21644dcd289a admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:28.877366 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1275/2541] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:58:28 2022] PUT /identity/v3/projects/ea41349cca9f410c942383d1b1acfada/users/78fa82aae45d447aba89cb63b8d843af/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:58:28.889331 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a1f526-2ebd-4c5c-bc63-21a39b88d8c8 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:28.889616 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a1f526-2ebd-4c5c-bc63-21a39b88d8c8 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:28.889891 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a1f526-2ebd-4c5c-bc63-21a39b88d8c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:28.917982 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-b3a1f526-2ebd-4c5c-bc63-21a39b88d8c8 None None] MFA Rules not processed for user `78fa82aae45d447aba89cb63b8d843af`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:58:28.919207 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-b3a1f526-2ebd-4c5c-bc63-21a39b88d8c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:28.965129 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:INSERT=4,keystone:DELETE=1,keystone:SELECT=95 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:58:28.986004 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1267/2542] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:58:28 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:58:28.997381 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94263e36-44a2-4524-9209-21644dcd289a admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:29.000133 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-94263e36-44a2-4524-9209-21644dcd289a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:29.058036 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94263e36-44a2-4524-9209-21644dcd289a admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:29.061606 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd981ff9-f07e-40d9-b821-fb622ede2278 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:29.062197 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd981ff9-f07e-40d9-b821-fb622ede2278 service cinder] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:29.062468 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd981ff9-f07e-40d9-b821-fb622ede2278 service cinder] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:29.062735 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd981ff9-f07e-40d9-b821-fb622ede2278 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:29.071325 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd981ff9-f07e-40d9-b821-fb622ede2278 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:29.072731 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd981ff9-f07e-40d9-b821-fb622ede2278 service cinder] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:29.081576 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1276/2543] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:58:28 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:58:29.104293 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a1f526-2ebd-4c5c-bc63-21a39b88d8c8 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:29.166958 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a1f526-2ebd-4c5c-bc63-21a39b88d8c8 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:29.171534 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96184ecc-4010-4171-9b32-5b4e0dc380ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:29.172156 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96184ecc-4010-4171-9b32-5b4e0dc380ba admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:29.172430 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96184ecc-4010-4171-9b32-5b4e0dc380ba admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:29.172698 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96184ecc-4010-4171-9b32-5b4e0dc380ba admin admin] PATH_INFO: `/v3/users/78fa82aae45d447aba89cb63b8d843af` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:29.173829 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96184ecc-4010-4171-9b32-5b4e0dc380ba admin admin] RBAC: Authorizing `identity:delete_user(user_id=78fa82aae45d447aba89cb63b8d843af)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:29.185864 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96184ecc-4010-4171-9b32-5b4e0dc380ba admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:29.237277 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-96184ecc-4010-4171-9b32-5b4e0dc380ba admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '78fa82aae45d447aba89cb63b8d843af'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:29.237860 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-96184ecc-4010-4171-9b32-5b4e0dc380ba admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '78fa82aae45d447aba89cb63b8d843af'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:29.238477 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-96184ecc-4010-4171-9b32-5b4e0dc380ba admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '78fa82aae45d447aba89cb63b8d843af'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:29.245431 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-96184ecc-4010-4171-9b32-5b4e0dc380ba admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '78fa82aae45d447aba89cb63b8d843af'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:29.261448 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-96184ecc-4010-4171-9b32-5b4e0dc380ba admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '78fa82aae45d447aba89cb63b8d843af'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:29.275046 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1268/2544] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:58:29 2022] DELETE /identity/v3/users/78fa82aae45d447aba89cb63b8d843af => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:58:29.279657 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c012a11-b8f6-469a-bfcd-dd65ae435e1c None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:29.279944 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c012a11-b8f6-469a-bfcd-dd65ae435e1c None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:29.280218 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c012a11-b8f6-469a-bfcd-dd65ae435e1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:29.300472 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-7c012a11-b8f6-469a-bfcd-dd65ae435e1c None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:58:29.301652 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-7c012a11-b8f6-469a-bfcd-dd65ae435e1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:29.329855 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:58:29.361468 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1277/2545] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:58:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:58:29.371501 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96184ecc-4010-4171-9b32-5b4e0dc380ba admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:29.372868 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-96184ecc-4010-4171-9b32-5b4e0dc380ba admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:29.431951 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96184ecc-4010-4171-9b32-5b4e0dc380ba admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:29.435658 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76476de6-695c-41ab-8bdc-d993bcc4f3fc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:29.436258 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76476de6-695c-41ab-8bdc-d993bcc4f3fc service neutron] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:29.436535 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76476de6-695c-41ab-8bdc-d993bcc4f3fc service neutron] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:29.436805 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76476de6-695c-41ab-8bdc-d993bcc4f3fc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:29.446328 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76476de6-695c-41ab-8bdc-d993bcc4f3fc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:29.447789 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76476de6-695c-41ab-8bdc-d993bcc4f3fc service neutron] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:29.457196 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1269/2546] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:58:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:58:29.788788 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c012a11-b8f6-469a-bfcd-dd65ae435e1c None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:29.790289 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-7c012a11-b8f6-469a-bfcd-dd65ae435e1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:29.804269 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c012a11-b8f6-469a-bfcd-dd65ae435e1c None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:29.808884 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-302f3df8-d00e-4ae0-a316-543d50f5ccae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:29.809514 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-302f3df8-d00e-4ae0-a316-543d50f5ccae admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:29.809786 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-302f3df8-d00e-4ae0-a316-543d50f5ccae admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:29.810049 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-302f3df8-d00e-4ae0-a316-543d50f5ccae admin admin] PATH_INFO: `/v3/projects/ea41349cca9f410c942383d1b1acfada` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:29.811830 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-302f3df8-d00e-4ae0-a316-543d50f5ccae admin admin] RBAC: Authorizing `identity:delete_project(project_id=ea41349cca9f410c942383d1b1acfada)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:29.813715 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-302f3df8-d00e-4ae0-a316-543d50f5ccae admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:29.859614 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-302f3df8-d00e-4ae0-a316-543d50f5ccae admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ea41349cca9f410c942383d1b1acfada', 'request_id': 'req-302f3df8-d00e-4ae0-a316-543d50f5ccae'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:29.865456 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-302f3df8-d00e-4ae0-a316-543d50f5ccae admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ea41349cca9f410c942383d1b1acfada', 'request_id': 'req-302f3df8-d00e-4ae0-a316-543d50f5ccae'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:29.873106 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-302f3df8-d00e-4ae0-a316-543d50f5ccae admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ea41349cca9f410c942383d1b1acfada', 'request_id': 'req-302f3df8-d00e-4ae0-a316-543d50f5ccae'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:29.883567 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-302f3df8-d00e-4ae0-a316-543d50f5ccae admin admin] The token cache is being invalidate because project ea41349cca9f410c942383d1b1acfada was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87924) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:58:29.883843 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-302f3df8-d00e-4ae0-a316-543d50f5ccae admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:29.884484 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-302f3df8-d00e-4ae0-a316-543d50f5ccae admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:29.885906 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1278/2547] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:58:29 2022] DELETE /identity/v3/projects/ea41349cca9f410c942383d1b1acfada => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:58:29.890715 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1caf2083-9181-4f84-a3b6-27532e4b5adc None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:29.891205 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1caf2083-9181-4f84-a3b6-27532e4b5adc None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:29.891302 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1caf2083-9181-4f84-a3b6-27532e4b5adc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:29.911367 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-1caf2083-9181-4f84-a3b6-27532e4b5adc None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:58:29.913166 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-1caf2083-9181-4f84-a3b6-27532e4b5adc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:29.917262 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:58:29.973279 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1270/2548] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:58:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:58:29.982817 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2bb8c6-0b37-4d21-8816-80fd6d08ba68 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:29.983114 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2bb8c6-0b37-4d21-8816-80fd6d08ba68 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:29.983386 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2bb8c6-0b37-4d21-8816-80fd6d08ba68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:30.002673 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-bd2bb8c6-0b37-4d21-8816-80fd6d08ba68 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:58:30.003896 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-bd2bb8c6-0b37-4d21-8816-80fd6d08ba68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:30.011985 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1279/2549] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:58:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:58:30.016269 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1caf2083-9181-4f84-a3b6-27532e4b5adc None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:30.026510 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1caf2083-9181-4f84-a3b6-27532e4b5adc None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:30.031218 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e9d422a-508e-4c48-95c3-4a4034bd596f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:30.031758 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e9d422a-508e-4c48-95c3-4a4034bd596f admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:30.032034 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e9d422a-508e-4c48-95c3-4a4034bd596f admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:30.032562 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e9d422a-508e-4c48-95c3-4a4034bd596f admin admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:30.033207 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e9d422a-508e-4c48-95c3-4a4034bd596f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:58:30.033507 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e9d422a-508e-4c48-95c3-4a4034bd596f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:30.035157 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e9d422a-508e-4c48-95c3-4a4034bd596f admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:30.047343 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1271/2550] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 00:58:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:58:30.051523 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd2bb8c6-0b37-4d21-8816-80fd6d08ba68 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:30.060672 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd2bb8c6-0b37-4d21-8816-80fd6d08ba68 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:30.065223 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1842577-13b8-482f-9c35-04c017e61954 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:30.065803 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1842577-13b8-482f-9c35-04c017e61954 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:30.066076 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1842577-13b8-482f-9c35-04c017e61954 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:30.066342 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1842577-13b8-482f-9c35-04c017e61954 admin admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:30.067035 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1842577-13b8-482f-9c35-04c017e61954 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:30.069082 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1842577-13b8-482f-9c35-04c017e61954 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:30.097354 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1280/2551] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:58:30 2022] POST /identity/v3/projects => generated 375 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:58:30.102258 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e9d422a-508e-4c48-95c3-4a4034bd596f admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:30.112591 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e9d422a-508e-4c48-95c3-4a4034bd596f admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:30.169610 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf5f1c04-f810-4356-89cf-116ba2ff9d75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:30.170213 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5f1c04-f810-4356-89cf-116ba2ff9d75 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:30.170490 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5f1c04-f810-4356-89cf-116ba2ff9d75 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:30.170763 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5f1c04-f810-4356-89cf-116ba2ff9d75 admin admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:30.171412 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf5f1c04-f810-4356-89cf-116ba2ff9d75 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:30.173325 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf5f1c04-f810-4356-89cf-116ba2ff9d75 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:30.194650 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1272/2552] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:58:30 2022] POST /identity/v3/users => generated 288 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:58:30.198773 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1842577-13b8-482f-9c35-04c017e61954 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:30.208733 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1842577-13b8-482f-9c35-04c017e61954 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:30.214123 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b068e8ce-a487-45f2-a8f5-85db442093bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:30.214710 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b068e8ce-a487-45f2-a8f5-85db442093bb admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:30.215018 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b068e8ce-a487-45f2-a8f5-85db442093bb admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:30.215287 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b068e8ce-a487-45f2-a8f5-85db442093bb admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:30.215915 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b068e8ce-a487-45f2-a8f5-85db442093bb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:30.217476 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b068e8ce-a487-45f2-a8f5-85db442093bb admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:30.227492 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1281/2553] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:58:30 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:58:30.231338 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf5f1c04-f810-4356-89cf-116ba2ff9d75 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:30.241194 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf5f1c04-f810-4356-89cf-116ba2ff9d75 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:30.245826 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb3c0559-3518-43f6-95ed-34d5bcb2d37e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:30.246412 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3c0559-3518-43f6-95ed-34d5bcb2d37e admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:30.246690 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3c0559-3518-43f6-95ed-34d5bcb2d37e admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:30.246996 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3c0559-3518-43f6-95ed-34d5bcb2d37e admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:30.247636 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb3c0559-3518-43f6-95ed-34d5bcb2d37e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:30.249219 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb3c0559-3518-43f6-95ed-34d5bcb2d37e admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:30.260487 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1273/2554] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:58:30 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:58:30.264672 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b068e8ce-a487-45f2-a8f5-85db442093bb admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:30.275202 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b068e8ce-a487-45f2-a8f5-85db442093bb admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:30.279746 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-605fd68c-2af7-493e-ad77-41f94506ba47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:30.280380 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-605fd68c-2af7-493e-ad77-41f94506ba47 admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:30.280676 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-605fd68c-2af7-493e-ad77-41f94506ba47 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:30.280944 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-605fd68c-2af7-493e-ad77-41f94506ba47 admin admin] PATH_INFO: `/v3/projects/ed97dc7408ea4b908e831dc309be4fc4/users/c3c78968b67248e99c06a3b6a4127136/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:30.295292 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-605fd68c-2af7-493e-ad77-41f94506ba47 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ed97dc7408ea4b908e831dc309be4fc4, user_id=c3c78968b67248e99c06a3b6a4127136, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:30.297808 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-605fd68c-2af7-493e-ad77-41f94506ba47 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:30.311240 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1282/2555] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:58:30 2022] PUT /identity/v3/projects/ed97dc7408ea4b908e831dc309be4fc4/users/c3c78968b67248e99c06a3b6a4127136/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:58:30.322852 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0506b61-c4fa-4562-9b20-909a5a475cdb None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:30.323141 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0506b61-c4fa-4562-9b20-909a5a475cdb None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:30.323412 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0506b61-c4fa-4562-9b20-909a5a475cdb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:30.352106 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-b0506b61-c4fa-4562-9b20-909a5a475cdb None None] MFA Rules not processed for user `c3c78968b67248e99c06a3b6a4127136`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:58:30.353305 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-b0506b61-c4fa-4562-9b20-909a5a475cdb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:30.417802 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1274/2556] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:58:30 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:58:30.422415 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-605fd68c-2af7-493e-ad77-41f94506ba47 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:30.478248 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:58:30.484765 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-605fd68c-2af7-493e-ad77-41f94506ba47 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:30.490318 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4548da98-18fc-496c-af34-643d157165c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:30.491025 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4548da98-18fc-496c-af34-643d157165c0 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:30.491403 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4548da98-18fc-496c-af34-643d157165c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:30.491678 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4548da98-18fc-496c-af34-643d157165c0 admin admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:30.492316 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4548da98-18fc-496c-af34-643d157165c0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:30.494188 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4548da98-18fc-496c-af34-643d157165c0 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:30.520071 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1283/2557] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:58:30 2022] POST /identity/v3/projects => generated 375 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:58:30.524783 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0506b61-c4fa-4562-9b20-909a5a475cdb None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:30.531286 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:58:30.535796 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0506b61-c4fa-4562-9b20-909a5a475cdb None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:30.594453 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e6ad356-b675-41cb-b36f-901390154f85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:30.595041 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6ad356-b675-41cb-b36f-901390154f85 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:30.595317 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6ad356-b675-41cb-b36f-901390154f85 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:30.595584 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6ad356-b675-41cb-b36f-901390154f85 admin admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:30.596218 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e6ad356-b675-41cb-b36f-901390154f85 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:30.598132 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e6ad356-b675-41cb-b36f-901390154f85 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:30.618458 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1275/2558] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:58:30 2022] POST /identity/v3/users => generated 294 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:58:30.622438 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4548da98-18fc-496c-af34-643d157165c0 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:30.633118 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4548da98-18fc-496c-af34-643d157165c0 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:30.637685 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fff11e0-1680-472b-8d17-d633cf0d0be8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:30.638276 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fff11e0-1680-472b-8d17-d633cf0d0be8 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:30.638558 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fff11e0-1680-472b-8d17-d633cf0d0be8 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:30.638890 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fff11e0-1680-472b-8d17-d633cf0d0be8 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:30.639494 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fff11e0-1680-472b-8d17-d633cf0d0be8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:30.641068 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fff11e0-1680-472b-8d17-d633cf0d0be8 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:30.651457 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1284/2559] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:58:30 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:58:30.655593 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e6ad356-b675-41cb-b36f-901390154f85 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:30.665970 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e6ad356-b675-41cb-b36f-901390154f85 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:30.670551 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db6e7e40-e840-4c2e-a9bd-f3c5dc54435d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:30.671218 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6e7e40-e840-4c2e-a9bd-f3c5dc54435d admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:30.671492 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6e7e40-e840-4c2e-a9bd-f3c5dc54435d admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:30.671759 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6e7e40-e840-4c2e-a9bd-f3c5dc54435d admin admin] PATH_INFO: `/v3/projects/23f529f3435e4a838f43a8f969736de4/users/3a6c2c0ff4cd4df6aced259b6a590b87/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:30.686898 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db6e7e40-e840-4c2e-a9bd-f3c5dc54435d admin admin] RBAC: Authorizing `identity:create_grant(project_id=23f529f3435e4a838f43a8f969736de4, user_id=3a6c2c0ff4cd4df6aced259b6a590b87, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:30.689533 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db6e7e40-e840-4c2e-a9bd-f3c5dc54435d admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:30.702943 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1276/2560] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:58:30 2022] PUT /identity/v3/projects/23f529f3435e4a838f43a8f969736de4/users/3a6c2c0ff4cd4df6aced259b6a590b87/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:58:30.706944 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fff11e0-1680-472b-8d17-d633cf0d0be8 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:30.717200 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fff11e0-1680-472b-8d17-d633cf0d0be8 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:30.774069 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdae0b82-9901-4313-a4d7-4aea4c29cb3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:30.774674 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdae0b82-9901-4313-a4d7-4aea4c29cb3e admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:30.774983 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdae0b82-9901-4313-a4d7-4aea4c29cb3e admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:30.775253 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdae0b82-9901-4313-a4d7-4aea4c29cb3e admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:30.775897 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdae0b82-9901-4313-a4d7-4aea4c29cb3e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:30.777479 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdae0b82-9901-4313-a4d7-4aea4c29cb3e admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:30.787893 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1285/2561] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:58:30 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:58:30.792324 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db6e7e40-e840-4c2e-a9bd-f3c5dc54435d admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:30.802551 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db6e7e40-e840-4c2e-a9bd-f3c5dc54435d admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:30.808307 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ba2de1-77ef-49fa-8fd7-8fbd149360e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:30.808957 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ba2de1-77ef-49fa-8fd7-8fbd149360e1 admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:30.809262 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ba2de1-77ef-49fa-8fd7-8fbd149360e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:30.809528 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ba2de1-77ef-49fa-8fd7-8fbd149360e1 admin admin] PATH_INFO: `/v3/projects/23f529f3435e4a838f43a8f969736de4/users/3a6c2c0ff4cd4df6aced259b6a590b87/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:30.811453 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ba2de1-77ef-49fa-8fd7-8fbd149360e1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=23f529f3435e4a838f43a8f969736de4, user_id=3a6c2c0ff4cd4df6aced259b6a590b87, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:30.814083 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ba2de1-77ef-49fa-8fd7-8fbd149360e1 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:30.825503 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1277/2562] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:58:30 2022] PUT /identity/v3/projects/23f529f3435e4a838f43a8f969736de4/users/3a6c2c0ff4cd4df6aced259b6a590b87/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:58:30.835796 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ab5b96-f023-40eb-844e-0ed77f6356c9 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:30.836291 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ab5b96-f023-40eb-844e-0ed77f6356c9 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:30.836403 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ab5b96-f023-40eb-844e-0ed77f6356c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:30.864272 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-e5ab5b96-f023-40eb-844e-0ed77f6356c9 None None] MFA Rules not processed for user `3a6c2c0ff4cd4df6aced259b6a590b87`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:58:30.865388 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-e5ab5b96-f023-40eb-844e-0ed77f6356c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:30.934828 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1286/2563] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:58:30 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:58:30.947518 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ba2de1-77ef-49fa-8fd7-8fbd149360e1 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:30.949304 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-04ba2de1-77ef-49fa-8fd7-8fbd149360e1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:31.007701 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ba2de1-77ef-49fa-8fd7-8fbd149360e1 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:31.011378 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91e4781b-2b28-4be9-8164-650bcaada427 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:31.011973 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e4781b-2b28-4be9-8164-650bcaada427 service cinder] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:31.012251 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e4781b-2b28-4be9-8164-650bcaada427 service cinder] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:31.012525 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e4781b-2b28-4be9-8164-650bcaada427 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:31.067357 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91e4781b-2b28-4be9-8164-650bcaada427 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:31.069366 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91e4781b-2b28-4be9-8164-650bcaada427 service cinder] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:31.078178 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1278/2564] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:58:30 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:58:33.626028 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5ab5b96-f023-40eb-844e-0ed77f6356c9 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:33.674992 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:58:33.687343 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5ab5b96-f023-40eb-844e-0ed77f6356c9 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:33.691760 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df7e5739-32bf-4403-9617-0dbff68426b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:33.692373 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7e5739-32bf-4403-9617-0dbff68426b4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:33.692648 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7e5739-32bf-4403-9617-0dbff68426b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:33.692915 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7e5739-32bf-4403-9617-0dbff68426b4 admin admin] PATH_INFO: `/v3/users/c3c78968b67248e99c06a3b6a4127136` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:33.694058 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df7e5739-32bf-4403-9617-0dbff68426b4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c3c78968b67248e99c06a3b6a4127136)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:33.695981 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df7e5739-32bf-4403-9617-0dbff68426b4 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:33.745724 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-df7e5739-32bf-4403-9617-0dbff68426b4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c3c78968b67248e99c06a3b6a4127136'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:33.746359 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-df7e5739-32bf-4403-9617-0dbff68426b4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c3c78968b67248e99c06a3b6a4127136'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:33.747069 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-df7e5739-32bf-4403-9617-0dbff68426b4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c3c78968b67248e99c06a3b6a4127136'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:33.753856 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-df7e5739-32bf-4403-9617-0dbff68426b4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c3c78968b67248e99c06a3b6a4127136'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:33.765945 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-df7e5739-32bf-4403-9617-0dbff68426b4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c3c78968b67248e99c06a3b6a4127136'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:33.778870 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1287/2565] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:58:33 2022] DELETE /identity/v3/users/c3c78968b67248e99c06a3b6a4127136 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:58:33.783682 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-687e81ba-2e29-4a0b-aa46-80c096ab9e91 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:33.784126 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-687e81ba-2e29-4a0b-aa46-80c096ab9e91 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:33.784236 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-687e81ba-2e29-4a0b-aa46-80c096ab9e91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:33.803621 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-687e81ba-2e29-4a0b-aa46-80c096ab9e91 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:58:33.804791 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-687e81ba-2e29-4a0b-aa46-80c096ab9e91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:33.810520 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:58:33.866455 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1279/2566] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:58:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:58:33.876378 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df7e5739-32bf-4403-9617-0dbff68426b4 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:33.877789 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-df7e5739-32bf-4403-9617-0dbff68426b4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:33.934704 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df7e5739-32bf-4403-9617-0dbff68426b4 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:33.938301 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-039e1948-2d96-424b-9e31-a9c7ec10f7ee service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:33.938929 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039e1948-2d96-424b-9e31-a9c7ec10f7ee service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:33.939206 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039e1948-2d96-424b-9e31-a9c7ec10f7ee service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:33.939472 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039e1948-2d96-424b-9e31-a9c7ec10f7ee service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:33.948806 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-039e1948-2d96-424b-9e31-a9c7ec10f7ee service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:33.950211 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-039e1948-2d96-424b-9e31-a9c7ec10f7ee service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:33.960113 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1288/2567] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:58:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:58:34.231039 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-687e81ba-2e29-4a0b-aa46-80c096ab9e91 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:34.232469 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-687e81ba-2e29-4a0b-aa46-80c096ab9e91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:34.244686 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-687e81ba-2e29-4a0b-aa46-80c096ab9e91 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:34.249670 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6860071e-2016-43ec-8433-6ab022aa56f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:34.250326 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6860071e-2016-43ec-8433-6ab022aa56f5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:34.250608 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6860071e-2016-43ec-8433-6ab022aa56f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:34.250919 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6860071e-2016-43ec-8433-6ab022aa56f5 admin admin] PATH_INFO: `/v3/projects/ed97dc7408ea4b908e831dc309be4fc4` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:34.252193 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6860071e-2016-43ec-8433-6ab022aa56f5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ed97dc7408ea4b908e831dc309be4fc4)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:34.254130 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6860071e-2016-43ec-8433-6ab022aa56f5 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:34.305017 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-6860071e-2016-43ec-8433-6ab022aa56f5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ed97dc7408ea4b908e831dc309be4fc4', 'request_id': 'req-6860071e-2016-43ec-8433-6ab022aa56f5'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:34.311621 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-6860071e-2016-43ec-8433-6ab022aa56f5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ed97dc7408ea4b908e831dc309be4fc4', 'request_id': 'req-6860071e-2016-43ec-8433-6ab022aa56f5'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:34.317391 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-6860071e-2016-43ec-8433-6ab022aa56f5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ed97dc7408ea4b908e831dc309be4fc4', 'request_id': 'req-6860071e-2016-43ec-8433-6ab022aa56f5'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:34.331343 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-6860071e-2016-43ec-8433-6ab022aa56f5 admin admin] The token cache is being invalidate because project ed97dc7408ea4b908e831dc309be4fc4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87923) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:58:34.331632 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-6860071e-2016-43ec-8433-6ab022aa56f5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:34.332396 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-6860071e-2016-43ec-8433-6ab022aa56f5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:34.334066 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1280/2568] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:58:34 2022] DELETE /identity/v3/projects/ed97dc7408ea4b908e831dc309be4fc4 => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:58:34.338672 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-039e1948-2d96-424b-9e31-a9c7ec10f7ee service neutron] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:34.348296 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-039e1948-2d96-424b-9e31-a9c7ec10f7ee service neutron] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:34.407015 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf43b55-8c07-4edd-b6d4-7a6a400257e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:34.407988 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf43b55-8c07-4edd-b6d4-7a6a400257e8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:34.408209 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf43b55-8c07-4edd-b6d4-7a6a400257e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:34.408476 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf43b55-8c07-4edd-b6d4-7a6a400257e8 admin admin] PATH_INFO: `/v3/users/3a6c2c0ff4cd4df6aced259b6a590b87` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:34.409601 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf43b55-8c07-4edd-b6d4-7a6a400257e8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3a6c2c0ff4cd4df6aced259b6a590b87)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:34.411547 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf43b55-8c07-4edd-b6d4-7a6a400257e8 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:34.426315 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:58:34.463247 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-4bf43b55-8c07-4edd-b6d4-7a6a400257e8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3a6c2c0ff4cd4df6aced259b6a590b87'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:34.463899 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-4bf43b55-8c07-4edd-b6d4-7a6a400257e8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3a6c2c0ff4cd4df6aced259b6a590b87'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:34.464505 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-4bf43b55-8c07-4edd-b6d4-7a6a400257e8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3a6c2c0ff4cd4df6aced259b6a590b87'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:34.471472 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-4bf43b55-8c07-4edd-b6d4-7a6a400257e8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3a6c2c0ff4cd4df6aced259b6a590b87'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:34.483307 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-4bf43b55-8c07-4edd-b6d4-7a6a400257e8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3a6c2c0ff4cd4df6aced259b6a590b87'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:34.495976 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1289/2569] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:58:34 2022] DELETE /identity/v3/users/3a6c2c0ff4cd4df6aced259b6a590b87 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:58:34.761700 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6860071e-2016-43ec-8433-6ab022aa56f5 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:34.763270 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-6860071e-2016-43ec-8433-6ab022aa56f5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:34.778132 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6860071e-2016-43ec-8433-6ab022aa56f5 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:34.835818 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f95c88f4-d002-4bff-8c1f-fa1ba67c8be5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:34.836516 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95c88f4-d002-4bff-8c1f-fa1ba67c8be5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:34.836711 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95c88f4-d002-4bff-8c1f-fa1ba67c8be5 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:34.836979 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95c88f4-d002-4bff-8c1f-fa1ba67c8be5 admin admin] PATH_INFO: `/v3/projects/23f529f3435e4a838f43a8f969736de4` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:34.838086 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f95c88f4-d002-4bff-8c1f-fa1ba67c8be5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=23f529f3435e4a838f43a8f969736de4)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:34.840009 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f95c88f4-d002-4bff-8c1f-fa1ba67c8be5 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:34.877098 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:58:34.886823 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-f95c88f4-d002-4bff-8c1f-fa1ba67c8be5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '23f529f3435e4a838f43a8f969736de4', 'request_id': 'req-f95c88f4-d002-4bff-8c1f-fa1ba67c8be5'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:34.892975 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-f95c88f4-d002-4bff-8c1f-fa1ba67c8be5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '23f529f3435e4a838f43a8f969736de4', 'request_id': 'req-f95c88f4-d002-4bff-8c1f-fa1ba67c8be5'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:34.898440 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-f95c88f4-d002-4bff-8c1f-fa1ba67c8be5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '23f529f3435e4a838f43a8f969736de4', 'request_id': 'req-f95c88f4-d002-4bff-8c1f-fa1ba67c8be5'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:34.911048 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-f95c88f4-d002-4bff-8c1f-fa1ba67c8be5 admin admin] The token cache is being invalidate because project 23f529f3435e4a838f43a8f969736de4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87923) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:58:34.911341 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-f95c88f4-d002-4bff-8c1f-fa1ba67c8be5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:34.911969 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-f95c88f4-d002-4bff-8c1f-fa1ba67c8be5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:34.913970 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1281/2570] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:58:34 2022] DELETE /identity/v3/projects/23f529f3435e4a838f43a8f969736de4 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:58:34.918892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96205265-42dc-4528-a518-6737c7f98416 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:34.919387 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96205265-42dc-4528-a518-6737c7f98416 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:34.919505 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96205265-42dc-4528-a518-6737c7f98416 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:34.939346 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-96205265-42dc-4528-a518-6737c7f98416 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:58:34.940466 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-96205265-42dc-4528-a518-6737c7f98416 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:34.998143 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1290/2571] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:58:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:58:35.062329 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69617eed-86e7-4f4a-a391-8e38f2c0a47f None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:35.062585 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69617eed-86e7-4f4a-a391-8e38f2c0a47f None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:35.062893 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69617eed-86e7-4f4a-a391-8e38f2c0a47f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:35.084720 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-69617eed-86e7-4f4a-a391-8e38f2c0a47f None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:58:35.085928 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-69617eed-86e7-4f4a-a391-8e38f2c0a47f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:35.093949 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1282/2572] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:58:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:58:35.098162 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96205265-42dc-4528-a518-6737c7f98416 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:35.108409 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96205265-42dc-4528-a518-6737c7f98416 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:35.112871 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab40ca7b-8c3b-497c-9568-0cf762f28626 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:35.113459 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab40ca7b-8c3b-497c-9568-0cf762f28626 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:35.113732 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab40ca7b-8c3b-497c-9568-0cf762f28626 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:35.113998 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab40ca7b-8c3b-497c-9568-0cf762f28626 admin admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:35.114633 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab40ca7b-8c3b-497c-9568-0cf762f28626 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:58:35.114965 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab40ca7b-8c3b-497c-9568-0cf762f28626 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:35.116850 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab40ca7b-8c3b-497c-9568-0cf762f28626 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:35.127971 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1291/2573] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 00:58:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:58:35.132471 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69617eed-86e7-4f4a-a391-8e38f2c0a47f None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:35.141641 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69617eed-86e7-4f4a-a391-8e38f2c0a47f None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:35.146241 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d45cb7b5-7f89-4d02-89a8-626c935440b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:35.147410 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45cb7b5-7f89-4d02-89a8-626c935440b7 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:35.147686 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45cb7b5-7f89-4d02-89a8-626c935440b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:35.147953 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45cb7b5-7f89-4d02-89a8-626c935440b7 admin admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:35.148575 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d45cb7b5-7f89-4d02-89a8-626c935440b7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:35.150472 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d45cb7b5-7f89-4d02-89a8-626c935440b7 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:35.178081 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1283/2574] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:58:35 2022] POST /identity/v3/projects => generated 385 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:58:35.182276 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab40ca7b-8c3b-497c-9568-0cf762f28626 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:35.192082 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab40ca7b-8c3b-497c-9568-0cf762f28626 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:35.248289 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e95da463-f6e8-4dad-8704-6916432ed27f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:35.248881 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95da463-f6e8-4dad-8704-6916432ed27f admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:35.249155 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95da463-f6e8-4dad-8704-6916432ed27f admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:35.249419 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95da463-f6e8-4dad-8704-6916432ed27f admin admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:35.250046 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e95da463-f6e8-4dad-8704-6916432ed27f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:35.251959 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e95da463-f6e8-4dad-8704-6916432ed27f admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:35.273995 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1292/2575] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:58:35 2022] POST /identity/v3/users => generated 293 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:58:35.278100 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d45cb7b5-7f89-4d02-89a8-626c935440b7 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:35.289510 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d45cb7b5-7f89-4d02-89a8-626c935440b7 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:35.294151 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a623a9-5a5c-41ae-9e7b-318a53429632 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:35.294742 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a623a9-5a5c-41ae-9e7b-318a53429632 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:35.295066 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a623a9-5a5c-41ae-9e7b-318a53429632 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:35.295334 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a623a9-5a5c-41ae-9e7b-318a53429632 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:35.295963 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18a623a9-5a5c-41ae-9e7b-318a53429632 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:35.297539 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18a623a9-5a5c-41ae-9e7b-318a53429632 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:35.307859 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1284/2576] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:58:35 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:58:35.311731 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e95da463-f6e8-4dad-8704-6916432ed27f admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:35.322060 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e95da463-f6e8-4dad-8704-6916432ed27f admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:35.326605 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-610bd0fd-a4de-4995-9178-3a56e22dec70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:35.327216 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610bd0fd-a4de-4995-9178-3a56e22dec70 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:35.327490 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610bd0fd-a4de-4995-9178-3a56e22dec70 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:35.327757 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610bd0fd-a4de-4995-9178-3a56e22dec70 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:35.328385 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-610bd0fd-a4de-4995-9178-3a56e22dec70 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:35.329949 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-610bd0fd-a4de-4995-9178-3a56e22dec70 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:35.331963 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=87,keystone:DELETE=9,keystone:INSERT=4 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:58:35.343962 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1293/2577] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:58:35 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:58:35.347347 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a623a9-5a5c-41ae-9e7b-318a53429632 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:35.358046 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a623a9-5a5c-41ae-9e7b-318a53429632 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:35.362661 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f6b168-20c2-44f1-b275-25fe84670742 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:35.363333 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f6b168-20c2-44f1-b275-25fe84670742 admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:35.363612 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f6b168-20c2-44f1-b275-25fe84670742 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:35.363884 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f6b168-20c2-44f1-b275-25fe84670742 admin admin] PATH_INFO: `/v3/projects/ad9fc47be36548c79bab892476bfbbfd/users/69819d8bd5e44bf8a9f7ddd6eefee232/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:35.378285 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13f6b168-20c2-44f1-b275-25fe84670742 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ad9fc47be36548c79bab892476bfbbfd, user_id=69819d8bd5e44bf8a9f7ddd6eefee232, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:35.380956 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13f6b168-20c2-44f1-b275-25fe84670742 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:35.393507 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1285/2578] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:58:35 2022] PUT /identity/v3/projects/ad9fc47be36548c79bab892476bfbbfd/users/69819d8bd5e44bf8a9f7ddd6eefee232/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:58:35.404686 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4b972e-54fd-459e-9842-c5e45ec71a36 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:35.404947 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4b972e-54fd-459e-9842-c5e45ec71a36 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:35.405213 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4b972e-54fd-459e-9842-c5e45ec71a36 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:35.432676 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-ef4b972e-54fd-459e-9842-c5e45ec71a36 None None] MFA Rules not processed for user `69819d8bd5e44bf8a9f7ddd6eefee232`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:58:35.433853 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-ef4b972e-54fd-459e-9842-c5e45ec71a36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:35.504022 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1294/2579] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:58:35 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:58:35.509191 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f6b168-20c2-44f1-b275-25fe84670742 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:35.574163 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f6b168-20c2-44f1-b275-25fe84670742 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:35.578807 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb78e66-2652-4efe-b772-91dba910d555 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:35.579408 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb78e66-2652-4efe-b772-91dba910d555 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:35.579685 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb78e66-2652-4efe-b772-91dba910d555 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:35.579961 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb78e66-2652-4efe-b772-91dba910d555 admin admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:35.580592 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efb78e66-2652-4efe-b772-91dba910d555 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:35.582510 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efb78e66-2652-4efe-b772-91dba910d555 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:35.612863 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1286/2580] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:58:35 2022] POST /identity/v3/projects => generated 385 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:58:35.616177 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4b972e-54fd-459e-9842-c5e45ec71a36 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:35.626923 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4b972e-54fd-459e-9842-c5e45ec71a36 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:35.681788 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09e0ee7a-d80d-4833-aeff-87fd903757f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:35.682384 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e0ee7a-d80d-4833-aeff-87fd903757f8 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:35.682661 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e0ee7a-d80d-4833-aeff-87fd903757f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:35.682961 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e0ee7a-d80d-4833-aeff-87fd903757f8 admin admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:35.683588 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09e0ee7a-d80d-4833-aeff-87fd903757f8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:35.685487 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09e0ee7a-d80d-4833-aeff-87fd903757f8 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:35.707017 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1295/2581] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:58:35 2022] POST /identity/v3/users => generated 293 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:58:35.711530 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb78e66-2652-4efe-b772-91dba910d555 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:35.721670 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb78e66-2652-4efe-b772-91dba910d555 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:35.726357 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22338871-b40a-4e33-9453-35c07948c9f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:35.727004 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22338871-b40a-4e33-9453-35c07948c9f6 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:35.727282 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22338871-b40a-4e33-9453-35c07948c9f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:35.727550 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22338871-b40a-4e33-9453-35c07948c9f6 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:35.728191 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22338871-b40a-4e33-9453-35c07948c9f6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:35.729789 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22338871-b40a-4e33-9453-35c07948c9f6 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:35.741504 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1287/2582] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:58:35 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:58:35.745262 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09e0ee7a-d80d-4833-aeff-87fd903757f8 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:35.754956 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09e0ee7a-d80d-4833-aeff-87fd903757f8 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:35.760145 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fbf62ee-712e-4ea4-93dd-1da70bb4d621 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:35.760728 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbf62ee-712e-4ea4-93dd-1da70bb4d621 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:35.761004 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbf62ee-712e-4ea4-93dd-1da70bb4d621 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:35.761274 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbf62ee-712e-4ea4-93dd-1da70bb4d621 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:35.761910 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fbf62ee-712e-4ea4-93dd-1da70bb4d621 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:35.763611 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fbf62ee-712e-4ea4-93dd-1da70bb4d621 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:35.774076 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1296/2583] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:58:35 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:58:35.778011 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22338871-b40a-4e33-9453-35c07948c9f6 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:35.787950 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22338871-b40a-4e33-9453-35c07948c9f6 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:35.792469 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-756c998c-2ff0-4dd8-acaa-5a74f89519aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:35.793094 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756c998c-2ff0-4dd8-acaa-5a74f89519aa admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:35.793364 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756c998c-2ff0-4dd8-acaa-5a74f89519aa admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:35.793626 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756c998c-2ff0-4dd8-acaa-5a74f89519aa admin admin] PATH_INFO: `/v3/projects/e5853fc69859466e83fca19876de5a22/users/bc3792cd5e5e4d6b86835fc4b37fd4b7/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:35.807741 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-756c998c-2ff0-4dd8-acaa-5a74f89519aa admin admin] RBAC: Authorizing `identity:create_grant(project_id=e5853fc69859466e83fca19876de5a22, user_id=bc3792cd5e5e4d6b86835fc4b37fd4b7, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:35.810381 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-756c998c-2ff0-4dd8-acaa-5a74f89519aa admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:35.823501 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1288/2584] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:58:35 2022] PUT /identity/v3/projects/e5853fc69859466e83fca19876de5a22/users/bc3792cd5e5e4d6b86835fc4b37fd4b7/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:58:35.835571 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5b8c8a-a95c-4d28-a852-9bd1134371a1 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:35.835929 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5b8c8a-a95c-4d28-a852-9bd1134371a1 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:35.836202 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5b8c8a-a95c-4d28-a852-9bd1134371a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:35.863183 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-ea5b8c8a-a95c-4d28-a852-9bd1134371a1 None None] MFA Rules not processed for user `bc3792cd5e5e4d6b86835fc4b37fd4b7`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:58:35.864394 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-ea5b8c8a-a95c-4d28-a852-9bd1134371a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:35.920000 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:58:35.930805 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1297/2585] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:58:35 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:58:35.935312 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-756c998c-2ff0-4dd8-acaa-5a74f89519aa admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:35.948430 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=91,keystone:DELETE=4,keystone:INSERT=5 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:58:35.998638 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-756c998c-2ff0-4dd8-acaa-5a74f89519aa admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:36.003196 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f294312-ddcd-4939-a868-9456aba9aca2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:36.003791 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f294312-ddcd-4939-a868-9456aba9aca2 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:36.004063 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f294312-ddcd-4939-a868-9456aba9aca2 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:36.004329 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f294312-ddcd-4939-a868-9456aba9aca2 admin admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:36.004958 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f294312-ddcd-4939-a868-9456aba9aca2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:36.007688 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f294312-ddcd-4939-a868-9456aba9aca2 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:36.033606 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1289/2586] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:58:35 2022] POST /identity/v3/projects => generated 385 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:58:36.037618 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea5b8c8a-a95c-4d28-a852-9bd1134371a1 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:36.047955 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea5b8c8a-a95c-4d28-a852-9bd1134371a1 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:36.102934 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-505161ed-9401-45d4-a790-2e131b2f5284 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:36.103544 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-505161ed-9401-45d4-a790-2e131b2f5284 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:36.103809 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-505161ed-9401-45d4-a790-2e131b2f5284 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:36.104550 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-505161ed-9401-45d4-a790-2e131b2f5284 admin admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:36.104675 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-505161ed-9401-45d4-a790-2e131b2f5284 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:36.106553 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-505161ed-9401-45d4-a790-2e131b2f5284 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:36.128058 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1298/2587] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:58:36 2022] POST /identity/v3/users => generated 299 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:58:36.132931 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f294312-ddcd-4939-a868-9456aba9aca2 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:36.143324 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f294312-ddcd-4939-a868-9456aba9aca2 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:36.147964 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b48af3d-dd30-4b6e-9400-6553cac9e8da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:36.148576 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b48af3d-dd30-4b6e-9400-6553cac9e8da admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:36.148888 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b48af3d-dd30-4b6e-9400-6553cac9e8da admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:36.149153 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b48af3d-dd30-4b6e-9400-6553cac9e8da admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:36.149793 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b48af3d-dd30-4b6e-9400-6553cac9e8da admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:36.151434 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b48af3d-dd30-4b6e-9400-6553cac9e8da admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:36.162301 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1290/2588] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:58:36 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:58:36.166515 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-505161ed-9401-45d4-a790-2e131b2f5284 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:36.176449 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-505161ed-9401-45d4-a790-2e131b2f5284 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:36.181920 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3625605a-3824-44df-8519-5fdee6a1329e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:36.182556 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3625605a-3824-44df-8519-5fdee6a1329e admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:36.182883 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3625605a-3824-44df-8519-5fdee6a1329e admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:36.183138 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3625605a-3824-44df-8519-5fdee6a1329e admin admin] PATH_INFO: `/v3/projects/ad5bf53f55924a32a28db426a9776fd9/users/fe687db427884494acf6c4b92a71d845/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:36.199068 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3625605a-3824-44df-8519-5fdee6a1329e admin admin] RBAC: Authorizing `identity:create_grant(project_id=ad5bf53f55924a32a28db426a9776fd9, user_id=fe687db427884494acf6c4b92a71d845, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:36.201680 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3625605a-3824-44df-8519-5fdee6a1329e admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:36.214514 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1299/2589] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:58:36 2022] PUT /identity/v3/projects/ad5bf53f55924a32a28db426a9776fd9/users/fe687db427884494acf6c4b92a71d845/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:58:36.218500 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b48af3d-dd30-4b6e-9400-6553cac9e8da admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:36.228779 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b48af3d-dd30-4b6e-9400-6553cac9e8da admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:36.283301 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8ca15a-8991-4e30-8e7a-5d33cf8dc756 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:36.283816 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8ca15a-8991-4e30-8e7a-5d33cf8dc756 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:36.284090 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8ca15a-8991-4e30-8e7a-5d33cf8dc756 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:36.284356 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8ca15a-8991-4e30-8e7a-5d33cf8dc756 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:36.284982 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a8ca15a-8991-4e30-8e7a-5d33cf8dc756 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:36.286560 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a8ca15a-8991-4e30-8e7a-5d33cf8dc756 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:36.296908 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1291/2590] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:58:36 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:58:36.300937 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3625605a-3824-44df-8519-5fdee6a1329e admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:36.313359 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3625605a-3824-44df-8519-5fdee6a1329e admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:36.320675 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5dd2336-7580-4a4b-ac77-5e65044afc3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:36.321330 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5dd2336-7580-4a4b-ac77-5e65044afc3d admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:36.321608 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5dd2336-7580-4a4b-ac77-5e65044afc3d admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:36.321880 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5dd2336-7580-4a4b-ac77-5e65044afc3d admin admin] PATH_INFO: `/v3/projects/ad5bf53f55924a32a28db426a9776fd9/users/fe687db427884494acf6c4b92a71d845/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:36.323868 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5dd2336-7580-4a4b-ac77-5e65044afc3d admin admin] RBAC: Authorizing `identity:create_grant(project_id=ad5bf53f55924a32a28db426a9776fd9, user_id=fe687db427884494acf6c4b92a71d845, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:36.326472 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5dd2336-7580-4a4b-ac77-5e65044afc3d admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:36.337090 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1300/2591] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:58:36 2022] PUT /identity/v3/projects/ad5bf53f55924a32a28db426a9776fd9/users/fe687db427884494acf6c4b92a71d845/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:58:36.347220 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb9629b-0e7a-4fd3-abf2-d328ff3ada1b None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:36.347507 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb9629b-0e7a-4fd3-abf2-d328ff3ada1b None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:36.347779 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb9629b-0e7a-4fd3-abf2-d328ff3ada1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:36.381139 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-9cb9629b-0e7a-4fd3-abf2-d328ff3ada1b None None] MFA Rules not processed for user `fe687db427884494acf6c4b92a71d845`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:58:36.382572 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-9cb9629b-0e7a-4fd3-abf2-d328ff3ada1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:36.450047 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1292/2592] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:58:36 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:58:36.461358 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5dd2336-7580-4a4b-ac77-5e65044afc3d admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:36.463057 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-e5dd2336-7580-4a4b-ac77-5e65044afc3d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:36.519611 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5dd2336-7580-4a4b-ac77-5e65044afc3d admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:36.523334 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfee4d34-1431-436d-bcf6-732bbd63a705 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:36.523926 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfee4d34-1431-436d-bcf6-732bbd63a705 service cinder] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:36.524201 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfee4d34-1431-436d-bcf6-732bbd63a705 service cinder] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:36.524470 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfee4d34-1431-436d-bcf6-732bbd63a705 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:36.581434 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfee4d34-1431-436d-bcf6-732bbd63a705 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:36.582819 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfee4d34-1431-436d-bcf6-732bbd63a705 service cinder] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:36.592426 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1301/2593] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:58:36 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 132 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 11 00:58:38.108963 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cb9629b-0e7a-4fd3-abf2-d328ff3ada1b None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:38.110384 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-9cb9629b-0e7a-4fd3-abf2-d328ff3ada1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:38.123909 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cb9629b-0e7a-4fd3-abf2-d328ff3ada1b None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:38.127540 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bed2a96-cff8-4d22-9251-4b986b2ae542 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:38.128165 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bed2a96-cff8-4d22-9251-4b986b2ae542 service cinder] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:38.128441 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bed2a96-cff8-4d22-9251-4b986b2ae542 service cinder] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:38.128710 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bed2a96-cff8-4d22-9251-4b986b2ae542 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:38.132888 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:58:38.187167 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bed2a96-cff8-4d22-9251-4b986b2ae542 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:38.188608 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bed2a96-cff8-4d22-9251-4b986b2ae542 service cinder] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:38.197687 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1293/2594] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:58:38 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:58:38.751699 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfee4d34-1431-436d-bcf6-732bbd63a705 service cinder] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:38.758172 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:58:38.761559 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfee4d34-1431-436d-bcf6-732bbd63a705 service cinder] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:38.766663 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57fa16f2-ef8d-4fa5-bd44-d02cc822bd59 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:38.767530 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57fa16f2-ef8d-4fa5-bd44-d02cc822bd59 service cinder] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:38.768069 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57fa16f2-ef8d-4fa5-bd44-d02cc822bd59 service cinder] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:38.768332 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57fa16f2-ef8d-4fa5-bd44-d02cc822bd59 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:38.778289 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57fa16f2-ef8d-4fa5-bd44-d02cc822bd59 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:38.779759 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57fa16f2-ef8d-4fa5-bd44-d02cc822bd59 service cinder] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:38.789848 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1302/2595] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:58:38 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:58:42.777023 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bed2a96-cff8-4d22-9251-4b986b2ae542 service cinder] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:42.853592 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bed2a96-cff8-4d22-9251-4b986b2ae542 service cinder] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:42.859848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8344efc0-83fe-4d0c-ba4c-d928002e48fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:42.860610 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8344efc0-83fe-4d0c-ba4c-d928002e48fe admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:42.861211 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8344efc0-83fe-4d0c-ba4c-d928002e48fe admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:42.861759 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8344efc0-83fe-4d0c-ba4c-d928002e48fe admin admin] PATH_INFO: `/v3/users/69819d8bd5e44bf8a9f7ddd6eefee232` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:42.864530 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8344efc0-83fe-4d0c-ba4c-d928002e48fe admin admin] RBAC: Authorizing `identity:delete_user(user_id=69819d8bd5e44bf8a9f7ddd6eefee232)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:42.866524 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8344efc0-83fe-4d0c-ba4c-d928002e48fe admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:42.927298 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-8344efc0-83fe-4d0c-ba4c-d928002e48fe admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '69819d8bd5e44bf8a9f7ddd6eefee232'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:42.927976 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-8344efc0-83fe-4d0c-ba4c-d928002e48fe admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '69819d8bd5e44bf8a9f7ddd6eefee232'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:42.928576 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-8344efc0-83fe-4d0c-ba4c-d928002e48fe admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '69819d8bd5e44bf8a9f7ddd6eefee232'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:42.939119 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-8344efc0-83fe-4d0c-ba4c-d928002e48fe admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '69819d8bd5e44bf8a9f7ddd6eefee232'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:42.952830 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-8344efc0-83fe-4d0c-ba4c-d928002e48fe admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '69819d8bd5e44bf8a9f7ddd6eefee232'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:42.972378 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1294/2596] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:58:42 2022] DELETE /identity/v3/users/69819d8bd5e44bf8a9f7ddd6eefee232 => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:58:42.973555 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971bc4d1-c695-4ea4-a3bc-ad22996ce850 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:42.974039 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971bc4d1-c695-4ea4-a3bc-ad22996ce850 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:42.974130 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971bc4d1-c695-4ea4-a3bc-ad22996ce850 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:42.995157 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-971bc4d1-c695-4ea4-a3bc-ad22996ce850 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:58:42.996669 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-971bc4d1-c695-4ea4-a3bc-ad22996ce850 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:43.066826 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1303/2597] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:58:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:58:43.077049 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8344efc0-83fe-4d0c-ba4c-d928002e48fe admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:43.078529 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-8344efc0-83fe-4d0c-ba4c-d928002e48fe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:43.109850 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:58:43.141209 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8344efc0-83fe-4d0c-ba4c-d928002e48fe admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:43.144835 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01faa37a-f791-475d-86ed-4efa6555dc98 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:43.145431 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01faa37a-f791-475d-86ed-4efa6555dc98 service neutron] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:43.145702 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01faa37a-f791-475d-86ed-4efa6555dc98 service neutron] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:43.145970 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01faa37a-f791-475d-86ed-4efa6555dc98 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:43.156205 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01faa37a-f791-475d-86ed-4efa6555dc98 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:43.157900 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01faa37a-f791-475d-86ed-4efa6555dc98 service neutron] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:43.167479 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1295/2598] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:58:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:58:43.381324 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2711b6b-d069-4041-b641-c5e31151b173 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:43.381607 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2711b6b-d069-4041-b641-c5e31151b173 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:43.381880 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2711b6b-d069-4041-b641-c5e31151b173 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:43.401438 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-d2711b6b-d069-4041-b641-c5e31151b173 None None] MFA Rules not processed for user `e1012f856881485cb40d916131a5d763`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:58:43.402644 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-d2711b6b-d069-4041-b641-c5e31151b173 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:43.435169 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01faa37a-f791-475d-86ed-4efa6555dc98 service neutron] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:43.436657 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-01faa37a-f791-475d-86ed-4efa6555dc98 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:43.450559 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01faa37a-f791-475d-86ed-4efa6555dc98 service neutron] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:43.455562 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d7eaa0-6a77-45ea-be16-e4b7554f2c37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:43.456188 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d7eaa0-6a77-45ea-be16-e4b7554f2c37 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:43.456467 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d7eaa0-6a77-45ea-be16-e4b7554f2c37 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:43.456740 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d7eaa0-6a77-45ea-be16-e4b7554f2c37 admin admin] PATH_INFO: `/v3/projects/ad9fc47be36548c79bab892476bfbbfd` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:43.457862 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d7eaa0-6a77-45ea-be16-e4b7554f2c37 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ad9fc47be36548c79bab892476bfbbfd)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:43.459808 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d7eaa0-6a77-45ea-be16-e4b7554f2c37 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:43.471252 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1304/2599] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:58:43 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:58:43.510011 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-a3d7eaa0-6a77-45ea-be16-e4b7554f2c37 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ad9fc47be36548c79bab892476bfbbfd', 'request_id': 'req-a3d7eaa0-6a77-45ea-be16-e4b7554f2c37'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:43.517958 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-a3d7eaa0-6a77-45ea-be16-e4b7554f2c37 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ad9fc47be36548c79bab892476bfbbfd', 'request_id': 'req-a3d7eaa0-6a77-45ea-be16-e4b7554f2c37'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:43.524463 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-a3d7eaa0-6a77-45ea-be16-e4b7554f2c37 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ad9fc47be36548c79bab892476bfbbfd', 'request_id': 'req-a3d7eaa0-6a77-45ea-be16-e4b7554f2c37'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:43.536718 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-a3d7eaa0-6a77-45ea-be16-e4b7554f2c37 admin admin] The token cache is being invalidate because project ad9fc47be36548c79bab892476bfbbfd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87923) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:58:43.537071 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-a3d7eaa0-6a77-45ea-be16-e4b7554f2c37 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:43.537770 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-a3d7eaa0-6a77-45ea-be16-e4b7554f2c37 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:43.539799 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1296/2600] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:58:43 2022] DELETE /identity/v3/projects/ad9fc47be36548c79bab892476bfbbfd => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:58:43.544235 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2711b6b-d069-4041-b641-c5e31151b173 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:43.557011 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2711b6b-d069-4041-b641-c5e31151b173 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:43.615861 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9048a209-31c8-4b88-905a-8766d437f690 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:43.616500 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9048a209-31c8-4b88-905a-8766d437f690 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:43.617277 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9048a209-31c8-4b88-905a-8766d437f690 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:43.617556 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9048a209-31c8-4b88-905a-8766d437f690 admin admin] PATH_INFO: `/v3/users/bc3792cd5e5e4d6b86835fc4b37fd4b7` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:43.618899 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9048a209-31c8-4b88-905a-8766d437f690 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bc3792cd5e5e4d6b86835fc4b37fd4b7)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:43.620852 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9048a209-31c8-4b88-905a-8766d437f690 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:43.660866 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:58:43.671321 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-9048a209-31c8-4b88-905a-8766d437f690 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bc3792cd5e5e4d6b86835fc4b37fd4b7'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:43.671985 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-9048a209-31c8-4b88-905a-8766d437f690 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bc3792cd5e5e4d6b86835fc4b37fd4b7'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:43.672606 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-9048a209-31c8-4b88-905a-8766d437f690 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bc3792cd5e5e4d6b86835fc4b37fd4b7'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:43.680409 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-9048a209-31c8-4b88-905a-8766d437f690 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bc3792cd5e5e4d6b86835fc4b37fd4b7'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:43.692563 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-9048a209-31c8-4b88-905a-8766d437f690 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bc3792cd5e5e4d6b86835fc4b37fd4b7'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:43.706213 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1305/2601] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:58:43 2022] DELETE /identity/v3/users/bc3792cd5e5e4d6b86835fc4b37fd4b7 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:58:43.818509 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d7eaa0-6a77-45ea-be16-e4b7554f2c37 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:43.821233 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-a3d7eaa0-6a77-45ea-be16-e4b7554f2c37 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:43.885523 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d7eaa0-6a77-45ea-be16-e4b7554f2c37 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:43.934794 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7f2ca0-aeb2-42fc-bf3b-f86e292dec3c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:43.935428 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7f2ca0-aeb2-42fc-bf3b-f86e292dec3c service neutron] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:43.935707 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7f2ca0-aeb2-42fc-bf3b-f86e292dec3c service neutron] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:43.935979 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7f2ca0-aeb2-42fc-bf3b-f86e292dec3c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:43.937753 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-bb7f2ca0-aeb2-42fc-bf3b-f86e292dec3c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:43.948339 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb7f2ca0-aeb2-42fc-bf3b-f86e292dec3c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:43.949796 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb7f2ca0-aeb2-42fc-bf3b-f86e292dec3c service neutron] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:43.957989 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1297/2602] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:58:43 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:58:43.996568 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9048a209-31c8-4b88-905a-8766d437f690 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:43.998180 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-9048a209-31c8-4b88-905a-8766d437f690 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:44.079111 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9048a209-31c8-4b88-905a-8766d437f690 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:44.085046 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dde840b-830b-4c6e-800a-52db162c4617 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:44.086013 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dde840b-830b-4c6e-800a-52db162c4617 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:44.086657 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dde840b-830b-4c6e-800a-52db162c4617 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:44.087289 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dde840b-830b-4c6e-800a-52db162c4617 admin admin] PATH_INFO: `/v3/projects/e5853fc69859466e83fca19876de5a22` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:44.088626 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dde840b-830b-4c6e-800a-52db162c4617 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e5853fc69859466e83fca19876de5a22)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:44.090909 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dde840b-830b-4c6e-800a-52db162c4617 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:44.142099 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-8dde840b-830b-4c6e-800a-52db162c4617 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e5853fc69859466e83fca19876de5a22', 'request_id': 'req-8dde840b-830b-4c6e-800a-52db162c4617'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:44.149015 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-8dde840b-830b-4c6e-800a-52db162c4617 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e5853fc69859466e83fca19876de5a22', 'request_id': 'req-8dde840b-830b-4c6e-800a-52db162c4617'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:44.159272 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-8dde840b-830b-4c6e-800a-52db162c4617 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e5853fc69859466e83fca19876de5a22', 'request_id': 'req-8dde840b-830b-4c6e-800a-52db162c4617'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:44.174167 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-8dde840b-830b-4c6e-800a-52db162c4617 admin admin] The token cache is being invalidate because project e5853fc69859466e83fca19876de5a22 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87924) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:58:44.174462 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-8dde840b-830b-4c6e-800a-52db162c4617 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:44.175180 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-8dde840b-830b-4c6e-800a-52db162c4617 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:44.180254 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1306/2603] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:58:43 2022] DELETE /identity/v3/projects/e5853fc69859466e83fca19876de5a22 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:58:44.181491 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7f2ca0-aeb2-42fc-bf3b-f86e292dec3c service neutron] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:44.195943 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7f2ca0-aeb2-42fc-bf3b-f86e292dec3c service neutron] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:44.201188 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:58:44.290702 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb134ed2-0b5e-4364-925f-532d4a133e78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:44.291425 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb134ed2-0b5e-4364-925f-532d4a133e78 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:44.291709 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb134ed2-0b5e-4364-925f-532d4a133e78 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:44.292885 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb134ed2-0b5e-4364-925f-532d4a133e78 admin admin] PATH_INFO: `/v3/users/fe687db427884494acf6c4b92a71d845` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:44.293827 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb134ed2-0b5e-4364-925f-532d4a133e78 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fe687db427884494acf6c4b92a71d845)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:44.295881 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb134ed2-0b5e-4364-925f-532d4a133e78 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:44.354763 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-cb134ed2-0b5e-4364-925f-532d4a133e78 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fe687db427884494acf6c4b92a71d845'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:44.355597 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-cb134ed2-0b5e-4364-925f-532d4a133e78 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fe687db427884494acf6c4b92a71d845'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:44.356587 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-cb134ed2-0b5e-4364-925f-532d4a133e78 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fe687db427884494acf6c4b92a71d845'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:44.366421 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-cb134ed2-0b5e-4364-925f-532d4a133e78 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fe687db427884494acf6c4b92a71d845'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:44.379986 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-cb134ed2-0b5e-4364-925f-532d4a133e78 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fe687db427884494acf6c4b92a71d845'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:44.400273 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1298/2604] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:58:44 2022] DELETE /identity/v3/users/fe687db427884494acf6c4b92a71d845 => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:58:44.739335 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dde840b-830b-4c6e-800a-52db162c4617 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:44.741016 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-8dde840b-830b-4c6e-800a-52db162c4617 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:44.777154 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dde840b-830b-4c6e-800a-52db162c4617 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:44.824567 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:58:44.844038 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c1edd42-1c35-4201-b42b-028f99c2a199 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:44.844667 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1edd42-1c35-4201-b42b-028f99c2a199 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:44.844963 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1edd42-1c35-4201-b42b-028f99c2a199 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:44.845228 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1edd42-1c35-4201-b42b-028f99c2a199 admin admin] PATH_INFO: `/v3/projects/ad5bf53f55924a32a28db426a9776fd9` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:44.846756 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c1edd42-1c35-4201-b42b-028f99c2a199 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ad5bf53f55924a32a28db426a9776fd9)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:44.848830 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c1edd42-1c35-4201-b42b-028f99c2a199 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:44.900962 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-2c1edd42-1c35-4201-b42b-028f99c2a199 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ad5bf53f55924a32a28db426a9776fd9', 'request_id': 'req-2c1edd42-1c35-4201-b42b-028f99c2a199'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:44.908323 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-2c1edd42-1c35-4201-b42b-028f99c2a199 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ad5bf53f55924a32a28db426a9776fd9', 'request_id': 'req-2c1edd42-1c35-4201-b42b-028f99c2a199'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:44.916993 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-2c1edd42-1c35-4201-b42b-028f99c2a199 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ad5bf53f55924a32a28db426a9776fd9', 'request_id': 'req-2c1edd42-1c35-4201-b42b-028f99c2a199'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:44.930993 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-2c1edd42-1c35-4201-b42b-028f99c2a199 admin admin] The token cache is being invalidate because project ad5bf53f55924a32a28db426a9776fd9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87924) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:58:44.931260 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-2c1edd42-1c35-4201-b42b-028f99c2a199 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:44.931897 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-2c1edd42-1c35-4201-b42b-028f99c2a199 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:58:44.933355 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1307/2605] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:58:44 2022] DELETE /identity/v3/projects/ad5bf53f55924a32a28db426a9776fd9 => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:58:44.939690 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5b0507-96f3-4137-9eca-db59194cce60 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:44.940175 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5b0507-96f3-4137-9eca-db59194cce60 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:44.940268 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5b0507-96f3-4137-9eca-db59194cce60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:44.961457 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-1e5b0507-96f3-4137-9eca-db59194cce60 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:58:44.962759 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-1e5b0507-96f3-4137-9eca-db59194cce60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:45.040863 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1299/2606] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:58:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:58:45.043380 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f33c55-ebec-46b9-9ef1-7e1f1a9d886c None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:45.043830 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f33c55-ebec-46b9-9ef1-7e1f1a9d886c None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:45.043928 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f33c55-ebec-46b9-9ef1-7e1f1a9d886c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:45.065774 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-f1f33c55-ebec-46b9-9ef1-7e1f1a9d886c None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:58:45.067002 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-f1f33c55-ebec-46b9-9ef1-7e1f1a9d886c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:45.076093 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1308/2607] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:58:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:58:45.080916 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5b0507-96f3-4137-9eca-db59194cce60 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:45.095457 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5b0507-96f3-4137-9eca-db59194cce60 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:45.100646 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13702dc2-5d47-44af-b4a4-48f9fb2f54e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:45.101397 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13702dc2-5d47-44af-b4a4-48f9fb2f54e2 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:45.101677 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13702dc2-5d47-44af-b4a4-48f9fb2f54e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:45.102040 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13702dc2-5d47-44af-b4a4-48f9fb2f54e2 admin admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:45.102704 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13702dc2-5d47-44af-b4a4-48f9fb2f54e2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:58:45.103076 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13702dc2-5d47-44af-b4a4-48f9fb2f54e2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:45.104878 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13702dc2-5d47-44af-b4a4-48f9fb2f54e2 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:45.117939 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1300/2608] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 00:58:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:58:45.122168 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f33c55-ebec-46b9-9ef1-7e1f1a9d886c None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:45.131499 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f33c55-ebec-46b9-9ef1-7e1f1a9d886c None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:45.136837 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93621c82-10ee-4a51-8d82-5c98dfa660f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:45.137427 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93621c82-10ee-4a51-8d82-5c98dfa660f3 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:45.137704 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93621c82-10ee-4a51-8d82-5c98dfa660f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:45.137971 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93621c82-10ee-4a51-8d82-5c98dfa660f3 admin admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:45.138603 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93621c82-10ee-4a51-8d82-5c98dfa660f3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:45.140519 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93621c82-10ee-4a51-8d82-5c98dfa660f3 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:45.174313 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1309/2609] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:58:45 2022] POST /identity/v3/projects => generated 381 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:58:45.175809 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13702dc2-5d47-44af-b4a4-48f9fb2f54e2 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:45.186801 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13702dc2-5d47-44af-b4a4-48f9fb2f54e2 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:45.203779 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:58:45.260562 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d9e97b7-cd3b-46ff-bcb0-4564f3b0387e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:45.261583 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9e97b7-cd3b-46ff-bcb0-4564f3b0387e admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:45.262023 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9e97b7-cd3b-46ff-bcb0-4564f3b0387e admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:45.262475 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9e97b7-cd3b-46ff-bcb0-4564f3b0387e admin admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:45.263162 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d9e97b7-cd3b-46ff-bcb0-4564f3b0387e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:45.265761 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d9e97b7-cd3b-46ff-bcb0-4564f3b0387e admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:45.291217 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1301/2610] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:58:45 2022] POST /identity/v3/users => generated 291 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:58:45.294753 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93621c82-10ee-4a51-8d82-5c98dfa660f3 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:45.313426 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93621c82-10ee-4a51-8d82-5c98dfa660f3 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:45.318888 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6570849-28fd-4bd7-8c16-ab3e33733bb5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:45.319524 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6570849-28fd-4bd7-8c16-ab3e33733bb5 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:45.319850 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6570849-28fd-4bd7-8c16-ab3e33733bb5 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:45.320147 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6570849-28fd-4bd7-8c16-ab3e33733bb5 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:45.320801 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6570849-28fd-4bd7-8c16-ab3e33733bb5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:45.330883 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6570849-28fd-4bd7-8c16-ab3e33733bb5 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:45.347075 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1310/2611] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:58:45 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:58:45.348568 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d9e97b7-cd3b-46ff-bcb0-4564f3b0387e admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:45.365402 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d9e97b7-cd3b-46ff-bcb0-4564f3b0387e admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:45.378194 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d095274e-7ed1-4915-b987-9c43bc1618d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:45.379235 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d095274e-7ed1-4915-b987-9c43bc1618d3 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:45.379463 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d095274e-7ed1-4915-b987-9c43bc1618d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:45.379735 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d095274e-7ed1-4915-b987-9c43bc1618d3 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:45.380829 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d095274e-7ed1-4915-b987-9c43bc1618d3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:45.382929 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d095274e-7ed1-4915-b987-9c43bc1618d3 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:45.406615 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1302/2612] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:58:45 2022] GET /identity/v3/roles => generated 1420 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:58:45.411220 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6570849-28fd-4bd7-8c16-ab3e33733bb5 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:45.424885 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6570849-28fd-4bd7-8c16-ab3e33733bb5 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:45.432299 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28e735d6-048c-4eb5-8ef1-6c326708b0d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:45.433526 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e735d6-048c-4eb5-8ef1-6c326708b0d1 admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:45.433912 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e735d6-048c-4eb5-8ef1-6c326708b0d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:45.434277 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e735d6-048c-4eb5-8ef1-6c326708b0d1 admin admin] PATH_INFO: `/v3/projects/de36b740b8234aca8317d5e4440decec/users/78a2199d3f1d4bc5b1f988ae06c0d609/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:45.450263 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3f5587-2d89-49a2-ad6e-302cd3d6e45e None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:45.450546 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3f5587-2d89-49a2-ad6e-302cd3d6e45e None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:45.450842 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3f5587-2d89-49a2-ad6e-302cd3d6e45e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:45.452102 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28e735d6-048c-4eb5-8ef1-6c326708b0d1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=de36b740b8234aca8317d5e4440decec, user_id=78a2199d3f1d4bc5b1f988ae06c0d609, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:45.454901 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28e735d6-048c-4eb5-8ef1-6c326708b0d1 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:45.470423 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1311/2613] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:58:45 2022] PUT /identity/v3/projects/de36b740b8234aca8317d5e4440decec/users/78a2199d3f1d4bc5b1f988ae06c0d609/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:58:45.473938 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-4e3f5587-2d89-49a2-ad6e-302cd3d6e45e None None] MFA Rules not processed for user `9fc4274836844ae4ad5d16d74c36f671`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:58:45.474852 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46074568-54f9-432b-beda-2f55ad70cfd5 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:45.475151 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46074568-54f9-432b-beda-2f55ad70cfd5 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:45.475278 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-4e3f5587-2d89-49a2-ad6e-302cd3d6e45e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:45.475428 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46074568-54f9-432b-beda-2f55ad70cfd5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:45.496580 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-46074568-54f9-432b-beda-2f55ad70cfd5 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:58:45.497910 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-46074568-54f9-432b-beda-2f55ad70cfd5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:45.538476 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1303/2614] 10.222.0.27 () {58 vars in 1009 bytes} [Sun Sep 11 00:58:45 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:58:45.554703 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e3f5587-2d89-49a2-ad6e-302cd3d6e45e None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:45.556132 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-4e3f5587-2d89-49a2-ad6e-302cd3d6e45e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:45.569534 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1312/2615] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:58:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:58:45.580539 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46074568-54f9-432b-beda-2f55ad70cfd5 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:45.581979 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-46074568-54f9-432b-beda-2f55ad70cfd5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:45.596494 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:58:45.631163 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e3f5587-2d89-49a2-ad6e-302cd3d6e45e None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:45.637436 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b8487c-f752-46e1-8062-9946905379a1 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bcb73dec73464a3abc4b774abb4f416f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:45.638220 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b8487c-f752-46e1-8062-9946905379a1 service swift] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:45.638473 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b8487c-f752-46e1-8062-9946905379a1 service swift] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:45.638832 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b8487c-f752-46e1-8062-9946905379a1 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:45.643788 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46074568-54f9-432b-beda-2f55ad70cfd5 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:45.648012 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7b8487c-f752-46e1-8062-9946905379a1 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:45.648199 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b19f5ea9-ee06-4a67-b0c1-bf327a35eb12 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:45.648725 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19f5ea9-ee06-4a67-b0c1-bf327a35eb12 service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:45.649042 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19f5ea9-ee06-4a67-b0c1-bf327a35eb12 service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:45.649279 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19f5ea9-ee06-4a67-b0c1-bf327a35eb12 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:45.650224 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7b8487c-f752-46e1-8062-9946905379a1 service swift] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:45.658946 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1304/2616] 10.222.0.27 () {60 vars in 1384 bytes} [Sun Sep 11 00:58:45 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 11 00:58:45.660475 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b19f5ea9-ee06-4a67-b0c1-bf327a35eb12 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:45.662728 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b19f5ea9-ee06-4a67-b0c1-bf327a35eb12 service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:45.677543 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1313/2617] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:58:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 11 00:58:46.586005 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25771f8-7960-44db-a3b2-e8792191afde None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:46.586648 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25771f8-7960-44db-a3b2-e8792191afde None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:46.586890 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25771f8-7960-44db-a3b2-e8792191afde None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:46.641799 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-c25771f8-7960-44db-a3b2-e8792191afde None None] MFA Rules not processed for user `78a2199d3f1d4bc5b1f988ae06c0d609`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:58:46.643437 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-c25771f8-7960-44db-a3b2-e8792191afde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:46.671787 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:58:46.800018 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1305/2618] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:58:46 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:58:46.811567 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b19f5ea9-ee06-4a67-b0c1-bf327a35eb12 service neutron] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:58:46.813314 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-b19f5ea9-ee06-4a67-b0c1-bf327a35eb12 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:46.943973 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b19f5ea9-ee06-4a67-b0c1-bf327a35eb12 service neutron] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:58:46.957914 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3177c90-e2c6-47b7-82f0-81cb4ee1191b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:58:46.958555 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3177c90-e2c6-47b7-82f0-81cb4ee1191b service cinder] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:46.959906 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3177c90-e2c6-47b7-82f0-81cb4ee1191b service cinder] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:46.961255 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3177c90-e2c6-47b7-82f0-81cb4ee1191b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:46.986675 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3177c90-e2c6-47b7-82f0-81cb4ee1191b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:58:46.988334 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3177c90-e2c6-47b7-82f0-81cb4ee1191b service cinder] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:58:47.019375 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1314/2619] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:58:46 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 208 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:58:48.592619 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8e104c-baee-4b3b-874e-2da3013e1dc3 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:58:48.593146 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8e104c-baee-4b3b-874e-2da3013e1dc3 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:58:48.593146 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8e104c-baee-4b3b-874e-2da3013e1dc3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:58:48.639235 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-1c8e104c-baee-4b3b-874e-2da3013e1dc3 None None] MFA Rules not processed for user `78a2199d3f1d4bc5b1f988ae06c0d609`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:58:48.641619 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-1c8e104c-baee-4b3b-874e-2da3013e1dc3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:58:48.650570 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1306/2620] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:58:48 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 64 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:58:57.005575 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=47 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:58:58.633096 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=22 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:59:04.432175 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3177c90-e2c6-47b7-82f0-81cb4ee1191b service cinder] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:04.433616 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-b3177c90-e2c6-47b7-82f0-81cb4ee1191b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:59:04.447518 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3177c90-e2c6-47b7-82f0-81cb4ee1191b service cinder] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:04.451984 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c02ce7-0494-45d8-9982-af211971b26f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:04.452641 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c02ce7-0494-45d8-9982-af211971b26f service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:04.453115 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c02ce7-0494-45d8-9982-af211971b26f service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:04.453732 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c02ce7-0494-45d8-9982-af211971b26f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:04.455625 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-b5c02ce7-0494-45d8-9982-af211971b26f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:59:04.466986 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5c02ce7-0494-45d8-9982-af211971b26f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:04.468834 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5c02ce7-0494-45d8-9982-af211971b26f service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:04.490983 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1315/2621] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:59:04 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 59 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:59:07.251254 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95517c0b-109d-4e6a-aef1-723a1959fcae None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:07.251781 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95517c0b-109d-4e6a-aef1-723a1959fcae None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:07.251781 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95517c0b-109d-4e6a-aef1-723a1959fcae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:07.273796 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-95517c0b-109d-4e6a-aef1-723a1959fcae None None] MFA Rules not processed for user `9fc4274836844ae4ad5d16d74c36f671`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:59:07.274307 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-95517c0b-109d-4e6a-aef1-723a1959fcae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:59:07.282859 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1307/2622] 10.222.0.27 () {58 vars in 1009 bytes} [Sun Sep 11 00:59:07 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:59:07.296402 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c02ce7-0494-45d8-9982-af211971b26f service neutron] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:07.322416 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c02ce7-0494-45d8-9982-af211971b26f service neutron] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:07.327549 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8116d075-58ea-4df5-9e7a-f6bfc97471f4 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bcb73dec73464a3abc4b774abb4f416f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:07.328152 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8116d075-58ea-4df5-9e7a-f6bfc97471f4 service swift] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:07.328635 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8116d075-58ea-4df5-9e7a-f6bfc97471f4 service swift] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:07.328908 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8116d075-58ea-4df5-9e7a-f6bfc97471f4 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:07.337561 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8116d075-58ea-4df5-9e7a-f6bfc97471f4 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:07.339188 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8116d075-58ea-4df5-9e7a-f6bfc97471f4 service swift] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:07.348133 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1316/2623] 10.222.0.27 () {60 vars in 1384 bytes} [Sun Sep 11 00:59:07 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 11 00:59:07.890049 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95517c0b-109d-4e6a-aef1-723a1959fcae None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:07.893553 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-95517c0b-109d-4e6a-aef1-723a1959fcae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:59:08.052590 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95517c0b-109d-4e6a-aef1-723a1959fcae None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:08.059882 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88838008-62c4-443e-a560-696a7bde0ddc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '835def0e6c6d4365a2767051509eb0d6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'ResellerAdmin', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:08.060541 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88838008-62c4-443e-a560-696a7bde0ddc service nova] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:08.060782 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88838008-62c4-443e-a560-696a7bde0ddc service nova] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:08.061086 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88838008-62c4-443e-a560-696a7bde0ddc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:08.062838 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-88838008-62c4-443e-a560-696a7bde0ddc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:59:08.084272 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88838008-62c4-443e-a560-696a7bde0ddc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:08.090726 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88838008-62c4-443e-a560-696a7bde0ddc service nova] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:08.123196 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1308/2624] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:59:07 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 234 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:59:11.584885 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8116d075-58ea-4df5-9e7a-f6bfc97471f4 service swift] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:11.597917 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8116d075-58ea-4df5-9e7a-f6bfc97471f4 service swift] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:11.602596 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d6399e-93ae-4100-a4e1-2bc6a6c690df service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:11.603450 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d6399e-93ae-4100-a4e1-2bc6a6c690df service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:11.604135 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d6399e-93ae-4100-a4e1-2bc6a6c690df service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:11.605129 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d6399e-93ae-4100-a4e1-2bc6a6c690df service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:11.607235 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-a6d6399e-93ae-4100-a4e1-2bc6a6c690df service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:59:11.625060 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6d6399e-93ae-4100-a4e1-2bc6a6c690df service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:11.626609 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6d6399e-93ae-4100-a4e1-2bc6a6c690df service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:11.644860 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1317/2625] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:59:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 60 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:59:15.691268 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88838008-62c4-443e-a560-696a7bde0ddc service nova] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:15.692776 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-88838008-62c4-443e-a560-696a7bde0ddc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:59:15.707228 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88838008-62c4-443e-a560-696a7bde0ddc service nova] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:15.712354 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cdbe92d-6ef8-4c6c-a7ee-cac227f20c90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:15.712975 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cdbe92d-6ef8-4c6c-a7ee-cac227f20c90 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:15.713249 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cdbe92d-6ef8-4c6c-a7ee-cac227f20c90 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:15.713534 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cdbe92d-6ef8-4c6c-a7ee-cac227f20c90 admin admin] PATH_INFO: `/v3/users/d886b168a8eb4424987555c073d71727` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:15.714661 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cdbe92d-6ef8-4c6c-a7ee-cac227f20c90 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d886b168a8eb4424987555c073d71727)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:15.716682 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cdbe92d-6ef8-4c6c-a7ee-cac227f20c90 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:15.763206 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-3cdbe92d-6ef8-4c6c-a7ee-cac227f20c90 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd886b168a8eb4424987555c073d71727'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:15.763856 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-3cdbe92d-6ef8-4c6c-a7ee-cac227f20c90 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd886b168a8eb4424987555c073d71727'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:15.764461 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-3cdbe92d-6ef8-4c6c-a7ee-cac227f20c90 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd886b168a8eb4424987555c073d71727'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:15.771258 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-3cdbe92d-6ef8-4c6c-a7ee-cac227f20c90 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd886b168a8eb4424987555c073d71727'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:15.783860 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-3cdbe92d-6ef8-4c6c-a7ee-cac227f20c90 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd886b168a8eb4424987555c073d71727'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:15.797857 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1309/2626] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:59:15 2022] DELETE /identity/v3/users/d886b168a8eb4424987555c073d71727 => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:59:15.978524 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d6399e-93ae-4100-a4e1-2bc6a6c690df service neutron] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:15.988971 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d6399e-93ae-4100-a4e1-2bc6a6c690df service neutron] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:16.049685 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f5b6a0-0ef2-43e5-ae0d-4b3c8b95c198 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:16.050328 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f5b6a0-0ef2-43e5-ae0d-4b3c8b95c198 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:16.050605 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f5b6a0-0ef2-43e5-ae0d-4b3c8b95c198 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:16.050917 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f5b6a0-0ef2-43e5-ae0d-4b3c8b95c198 admin admin] PATH_INFO: `/v3/projects/172e557567404e0f906fd1c3452148d8` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:16.052065 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f5b6a0-0ef2-43e5-ae0d-4b3c8b95c198 admin admin] RBAC: Authorizing `identity:delete_project(project_id=172e557567404e0f906fd1c3452148d8)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:16.054545 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f5b6a0-0ef2-43e5-ae0d-4b3c8b95c198 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:16.109377 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-f9f5b6a0-0ef2-43e5-ae0d-4b3c8b95c198 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '172e557567404e0f906fd1c3452148d8', 'request_id': 'req-f9f5b6a0-0ef2-43e5-ae0d-4b3c8b95c198'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:16.116175 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-f9f5b6a0-0ef2-43e5-ae0d-4b3c8b95c198 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '172e557567404e0f906fd1c3452148d8', 'request_id': 'req-f9f5b6a0-0ef2-43e5-ae0d-4b3c8b95c198'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:16.123921 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-f9f5b6a0-0ef2-43e5-ae0d-4b3c8b95c198 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '172e557567404e0f906fd1c3452148d8', 'request_id': 'req-f9f5b6a0-0ef2-43e5-ae0d-4b3c8b95c198'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:16.134608 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-f9f5b6a0-0ef2-43e5-ae0d-4b3c8b95c198 admin admin] The token cache is being invalidate because project 172e557567404e0f906fd1c3452148d8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87924) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:59:16.134967 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-f9f5b6a0-0ef2-43e5-ae0d-4b3c8b95c198 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:16.135629 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-f9f5b6a0-0ef2-43e5-ae0d-4b3c8b95c198 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:16.137063 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1318/2627] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:59:15 2022] DELETE /identity/v3/projects/172e557567404e0f906fd1c3452148d8 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:59:16.141858 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb230dfe-41fa-4ac2-90e2-4753da7a1e9f None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:16.142146 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb230dfe-41fa-4ac2-90e2-4753da7a1e9f None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:16.142419 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb230dfe-41fa-4ac2-90e2-4753da7a1e9f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:16.164833 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-eb230dfe-41fa-4ac2-90e2-4753da7a1e9f None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:59:16.166051 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-eb230dfe-41fa-4ac2-90e2-4753da7a1e9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:59:16.216123 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:59:16.237123 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1310/2628] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:59:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:59:16.245754 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67934a0d-6ebe-41d7-b55a-39757727f117 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:16.246040 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67934a0d-6ebe-41d7-b55a-39757727f117 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:16.246315 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67934a0d-6ebe-41d7-b55a-39757727f117 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:16.267566 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-67934a0d-6ebe-41d7-b55a-39757727f117 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:59:16.269337 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-67934a0d-6ebe-41d7-b55a-39757727f117 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:59:16.280505 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1319/2629] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:59:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:59:16.285256 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb230dfe-41fa-4ac2-90e2-4753da7a1e9f None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:16.296550 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb230dfe-41fa-4ac2-90e2-4753da7a1e9f None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:16.313134 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3f4aac9-fce1-41bb-beee-d009bced441a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:16.313134 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3f4aac9-fce1-41bb-beee-d009bced441a admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:16.313516 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3f4aac9-fce1-41bb-beee-d009bced441a admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:16.313629 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3f4aac9-fce1-41bb-beee-d009bced441a admin admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:16.314221 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3f4aac9-fce1-41bb-beee-d009bced441a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:59:16.314605 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3f4aac9-fce1-41bb-beee-d009bced441a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:16.316161 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3f4aac9-fce1-41bb-beee-d009bced441a admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:16.328768 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1311/2630] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 00:59:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:59:16.333311 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67934a0d-6ebe-41d7-b55a-39757727f117 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:16.342980 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67934a0d-6ebe-41d7-b55a-39757727f117 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:16.347619 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69361f54-ab4d-423c-8c12-face5477890e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:16.348205 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69361f54-ab4d-423c-8c12-face5477890e admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:16.348479 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69361f54-ab4d-423c-8c12-face5477890e admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:16.348747 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69361f54-ab4d-423c-8c12-face5477890e admin admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:16.349370 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69361f54-ab4d-423c-8c12-face5477890e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:16.352031 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69361f54-ab4d-423c-8c12-face5477890e admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:16.378687 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1320/2631] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:59:16 2022] POST /identity/v3/projects => generated 383 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:59:16.383178 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3f4aac9-fce1-41bb-beee-d009bced441a admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:16.393259 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3f4aac9-fce1-41bb-beee-d009bced441a admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:16.449046 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a883666-fcd5-4398-8e52-f2908ed1b515 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:16.449574 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a883666-fcd5-4398-8e52-f2908ed1b515 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:16.449856 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a883666-fcd5-4398-8e52-f2908ed1b515 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:16.450128 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a883666-fcd5-4398-8e52-f2908ed1b515 admin admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:16.450814 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a883666-fcd5-4398-8e52-f2908ed1b515 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:16.452731 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a883666-fcd5-4398-8e52-f2908ed1b515 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:16.478877 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1312/2632] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:59:16 2022] POST /identity/v3/users => generated 292 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:59:16.480032 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69361f54-ab4d-423c-8c12-face5477890e admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:16.490625 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69361f54-ab4d-423c-8c12-face5477890e admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:16.495435 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfaff5fa-b7cc-4726-a793-37f18d89331c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:16.496183 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfaff5fa-b7cc-4726-a793-37f18d89331c admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:16.496532 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfaff5fa-b7cc-4726-a793-37f18d89331c admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:16.496873 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfaff5fa-b7cc-4726-a793-37f18d89331c admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:16.497587 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfaff5fa-b7cc-4726-a793-37f18d89331c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:16.499282 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfaff5fa-b7cc-4726-a793-37f18d89331c admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:16.512675 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1321/2633] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:59:16 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:59:16.513852 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a883666-fcd5-4398-8e52-f2908ed1b515 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:16.524516 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a883666-fcd5-4398-8e52-f2908ed1b515 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:16.529165 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2a111ac-a0ed-46cc-abd1-641d6d5d6cf4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:16.529757 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a111ac-a0ed-46cc-abd1-641d6d5d6cf4 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:16.530035 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a111ac-a0ed-46cc-abd1-641d6d5d6cf4 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:16.530307 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a111ac-a0ed-46cc-abd1-641d6d5d6cf4 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:16.530980 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2a111ac-a0ed-46cc-abd1-641d6d5d6cf4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:16.532828 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2a111ac-a0ed-46cc-abd1-641d6d5d6cf4 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:16.544208 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1313/2634] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:59:16 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:59:16.548233 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfaff5fa-b7cc-4726-a793-37f18d89331c admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:16.558250 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfaff5fa-b7cc-4726-a793-37f18d89331c admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:16.562745 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d81e4fb8-5bd2-4d37-9d8d-317577a3e8f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:16.563411 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d81e4fb8-5bd2-4d37-9d8d-317577a3e8f0 admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:16.563681 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d81e4fb8-5bd2-4d37-9d8d-317577a3e8f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:16.563944 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d81e4fb8-5bd2-4d37-9d8d-317577a3e8f0 admin admin] PATH_INFO: `/v3/projects/3de207e316eb493686e4d1b7592e94d2/users/52cf4afdd2c741f8800360522a6d2f01/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:16.572493 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:59:16.582367 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d81e4fb8-5bd2-4d37-9d8d-317577a3e8f0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3de207e316eb493686e4d1b7592e94d2, user_id=52cf4afdd2c741f8800360522a6d2f01, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:16.586403 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d81e4fb8-5bd2-4d37-9d8d-317577a3e8f0 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:16.599025 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1322/2635] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:59:16 2022] PUT /identity/v3/projects/3de207e316eb493686e4d1b7592e94d2/users/52cf4afdd2c741f8800360522a6d2f01/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:59:16.610103 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce5364b-c292-4b7a-bb87-28010646896e None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:16.610389 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce5364b-c292-4b7a-bb87-28010646896e None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:16.610660 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce5364b-c292-4b7a-bb87-28010646896e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:16.637874 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-8ce5364b-c292-4b7a-bb87-28010646896e None None] MFA Rules not processed for user `52cf4afdd2c741f8800360522a6d2f01`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:59:16.639111 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-8ce5364b-c292-4b7a-bb87-28010646896e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:59:16.703200 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1314/2636] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:59:16 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:59:16.714776 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d81e4fb8-5bd2-4d37-9d8d-317577a3e8f0 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:16.716454 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-d81e4fb8-5bd2-4d37-9d8d-317577a3e8f0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:59:16.788796 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d81e4fb8-5bd2-4d37-9d8d-317577a3e8f0 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:16.794358 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be609678-9231-49c4-bbbd-ce1d81bde5e2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '835def0e6c6d4365a2767051509eb0d6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'ResellerAdmin', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:16.794988 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be609678-9231-49c4-bbbd-ce1d81bde5e2 service nova] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:16.795269 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be609678-9231-49c4-bbbd-ce1d81bde5e2 service nova] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:16.795540 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be609678-9231-49c4-bbbd-ce1d81bde5e2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:16.803432 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be609678-9231-49c4-bbbd-ce1d81bde5e2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:16.805028 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be609678-9231-49c4-bbbd-ce1d81bde5e2 service nova] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:16.813722 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1323/2637] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:59:16 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:59:25.201380 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce5364b-c292-4b7a-bb87-28010646896e None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:25.246859 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:59:25.262673 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce5364b-c292-4b7a-bb87-28010646896e None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:25.267196 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a096a01-9102-400e-9538-8fb1e65c5265 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:25.267822 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a096a01-9102-400e-9538-8fb1e65c5265 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:25.268092 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a096a01-9102-400e-9538-8fb1e65c5265 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:25.268362 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a096a01-9102-400e-9538-8fb1e65c5265 admin admin] PATH_INFO: `/v3/users/52cf4afdd2c741f8800360522a6d2f01` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:25.269503 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a096a01-9102-400e-9538-8fb1e65c5265 admin admin] RBAC: Authorizing `identity:delete_user(user_id=52cf4afdd2c741f8800360522a6d2f01)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:25.272054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a096a01-9102-400e-9538-8fb1e65c5265 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:25.327180 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-0a096a01-9102-400e-9538-8fb1e65c5265 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '52cf4afdd2c741f8800360522a6d2f01'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:25.327180 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-0a096a01-9102-400e-9538-8fb1e65c5265 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '52cf4afdd2c741f8800360522a6d2f01'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:25.327180 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-0a096a01-9102-400e-9538-8fb1e65c5265 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '52cf4afdd2c741f8800360522a6d2f01'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:25.332062 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-0a096a01-9102-400e-9538-8fb1e65c5265 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '52cf4afdd2c741f8800360522a6d2f01'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:25.345033 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-0a096a01-9102-400e-9538-8fb1e65c5265 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '52cf4afdd2c741f8800360522a6d2f01'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:25.358351 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de72ba1-62fc-4fc6-84ba-807e30e7d28d None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:25.358877 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de72ba1-62fc-4fc6-84ba-807e30e7d28d None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:25.359067 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de72ba1-62fc-4fc6-84ba-807e30e7d28d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:25.359854 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1315/2638] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:59:25 2022] DELETE /identity/v3/users/52cf4afdd2c741f8800360522a6d2f01 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:59:25.364228 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60252a5e-833c-4c69-bc88-228178ce2801 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:25.364570 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60252a5e-833c-4c69-bc88-228178ce2801 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:25.365410 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60252a5e-833c-4c69-bc88-228178ce2801 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:25.380018 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-3de72ba1-62fc-4fc6-84ba-807e30e7d28d None None] MFA Rules not processed for user `9fc4274836844ae4ad5d16d74c36f671`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:59:25.382022 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-3de72ba1-62fc-4fc6-84ba-807e30e7d28d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:59:25.385289 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-60252a5e-833c-4c69-bc88-228178ce2801 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:59:25.386694 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-60252a5e-833c-4c69-bc88-228178ce2801 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:59:25.438903 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1324/2639] 10.222.0.27 () {58 vars in 967 bytes} [Sun Sep 11 00:59:25 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:59:25.447754 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1316/2640] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:59:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:59:25.457130 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de72ba1-62fc-4fc6-84ba-807e30e7d28d None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:25.458363 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-3de72ba1-62fc-4fc6-84ba-807e30e7d28d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:59:25.462345 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60252a5e-833c-4c69-bc88-228178ce2801 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:25.463966 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-60252a5e-833c-4c69-bc88-228178ce2801 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:59:25.515986 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de72ba1-62fc-4fc6-84ba-807e30e7d28d None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:25.519840 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60fa3fc5-77c0-4664-a70c-e9b39f2316ee service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:25.520449 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60fa3fc5-77c0-4664-a70c-e9b39f2316ee service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:25.520733 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60fa3fc5-77c0-4664-a70c-e9b39f2316ee service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:25.521550 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60fa3fc5-77c0-4664-a70c-e9b39f2316ee service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:25.530854 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60fa3fc5-77c0-4664-a70c-e9b39f2316ee service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:25.532696 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60fa3fc5-77c0-4664-a70c-e9b39f2316ee service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:25.534969 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60252a5e-833c-4c69-bc88-228178ce2801 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:25.540719 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046b1160-59a3-41f7-be0c-01f11b009e34 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bcb73dec73464a3abc4b774abb4f416f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:25.541342 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046b1160-59a3-41f7-be0c-01f11b009e34 service swift] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:25.541612 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046b1160-59a3-41f7-be0c-01f11b009e34 service swift] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:25.541875 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046b1160-59a3-41f7-be0c-01f11b009e34 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:25.543130 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1325/2641] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:59:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:59:25.548540 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:59:25.551036 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-046b1160-59a3-41f7-be0c-01f11b009e34 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:25.553642 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-046b1160-59a3-41f7-be0c-01f11b009e34 service swift] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:25.563446 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1317/2642] 10.222.0.27 () {60 vars in 1384 bytes} [Sun Sep 11 00:59:25 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 11 00:59:25.885646 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60fa3fc5-77c0-4664-a70c-e9b39f2316ee service neutron] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:25.887563 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-60fa3fc5-77c0-4664-a70c-e9b39f2316ee service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:59:25.912875 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60fa3fc5-77c0-4664-a70c-e9b39f2316ee service neutron] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:25.920873 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a42560-3bf3-48a2-a081-7c6752a05f12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:25.921452 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a42560-3bf3-48a2-a081-7c6752a05f12 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:25.921730 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a42560-3bf3-48a2-a081-7c6752a05f12 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:25.922064 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a42560-3bf3-48a2-a081-7c6752a05f12 admin admin] PATH_INFO: `/v3/projects/3de207e316eb493686e4d1b7592e94d2` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:25.923371 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a42560-3bf3-48a2-a081-7c6752a05f12 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3de207e316eb493686e4d1b7592e94d2)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:25.925360 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a42560-3bf3-48a2-a081-7c6752a05f12 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:25.943655 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:59:25.975777 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-a4a42560-3bf3-48a2-a081-7c6752a05f12 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3de207e316eb493686e4d1b7592e94d2', 'request_id': 'req-a4a42560-3bf3-48a2-a081-7c6752a05f12'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:25.982010 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-a4a42560-3bf3-48a2-a081-7c6752a05f12 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3de207e316eb493686e4d1b7592e94d2', 'request_id': 'req-a4a42560-3bf3-48a2-a081-7c6752a05f12'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:25.988845 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-a4a42560-3bf3-48a2-a081-7c6752a05f12 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3de207e316eb493686e4d1b7592e94d2', 'request_id': 'req-a4a42560-3bf3-48a2-a081-7c6752a05f12'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:26.000591 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-a4a42560-3bf3-48a2-a081-7c6752a05f12 admin admin] The token cache is being invalidate because project 3de207e316eb493686e4d1b7592e94d2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87924) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:59:26.001134 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-a4a42560-3bf3-48a2-a081-7c6752a05f12 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:26.001850 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-a4a42560-3bf3-48a2-a081-7c6752a05f12 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:26.003555 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1326/2643] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:59:25 2022] DELETE /identity/v3/projects/3de207e316eb493686e4d1b7592e94d2 => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:59:26.009157 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4f04b73-3579-49b0-8647-f7bc2ff70ac5 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:26.009442 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4f04b73-3579-49b0-8647-f7bc2ff70ac5 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:26.009713 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4f04b73-3579-49b0-8647-f7bc2ff70ac5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:26.029721 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-c4f04b73-3579-49b0-8647-f7bc2ff70ac5 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:59:26.030961 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-c4f04b73-3579-49b0-8647-f7bc2ff70ac5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:59:26.092311 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1318/2644] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:59:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:59:26.101289 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c50966a-40bd-4137-815f-26ab5b9fab97 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:26.101710 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c50966a-40bd-4137-815f-26ab5b9fab97 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:26.102063 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c50966a-40bd-4137-815f-26ab5b9fab97 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:26.123995 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-0c50966a-40bd-4137-815f-26ab5b9fab97 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:59:26.125214 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-0c50966a-40bd-4137-815f-26ab5b9fab97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:59:26.135262 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1327/2645] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:59:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:59:26.140538 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4f04b73-3579-49b0-8647-f7bc2ff70ac5 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:26.151593 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4f04b73-3579-49b0-8647-f7bc2ff70ac5 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:26.157751 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030ef45a-a6cd-4317-abd9-728b470e5cc4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:26.158774 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030ef45a-a6cd-4317-abd9-728b470e5cc4 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:26.159082 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030ef45a-a6cd-4317-abd9-728b470e5cc4 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:26.159349 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030ef45a-a6cd-4317-abd9-728b470e5cc4 admin admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:26.159998 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-030ef45a-a6cd-4317-abd9-728b470e5cc4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:59:26.160299 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-030ef45a-a6cd-4317-abd9-728b470e5cc4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:26.161897 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-030ef45a-a6cd-4317-abd9-728b470e5cc4 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:26.174868 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1319/2646] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 00:59:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:59:26.179920 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c50966a-40bd-4137-815f-26ab5b9fab97 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:26.188975 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c50966a-40bd-4137-815f-26ab5b9fab97 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:26.195284 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f880eb-d3fb-43b1-a14b-2bb94fe44255 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:26.195882 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f880eb-d3fb-43b1-a14b-2bb94fe44255 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:26.196155 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f880eb-d3fb-43b1-a14b-2bb94fe44255 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:26.196420 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f880eb-d3fb-43b1-a14b-2bb94fe44255 admin admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:26.197195 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33f880eb-d3fb-43b1-a14b-2bb94fe44255 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:26.199227 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33f880eb-d3fb-43b1-a14b-2bb94fe44255 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:26.225849 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1328/2647] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:59:26 2022] POST /identity/v3/projects => generated 365 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:59:26.230228 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030ef45a-a6cd-4317-abd9-728b470e5cc4 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:26.240815 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030ef45a-a6cd-4317-abd9-728b470e5cc4 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:26.297107 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f2f2f66-14b7-4d6b-9a18-4c015ae7edeb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:26.297703 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2f2f66-14b7-4d6b-9a18-4c015ae7edeb admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:26.297981 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2f2f66-14b7-4d6b-9a18-4c015ae7edeb admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:26.298250 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2f2f66-14b7-4d6b-9a18-4c015ae7edeb admin admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:26.298911 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f2f2f66-14b7-4d6b-9a18-4c015ae7edeb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:26.300832 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f2f2f66-14b7-4d6b-9a18-4c015ae7edeb admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:26.321418 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1320/2648] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:59:26 2022] POST /identity/v3/users => generated 283 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:59:26.325555 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f880eb-d3fb-43b1-a14b-2bb94fe44255 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:26.336669 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f880eb-d3fb-43b1-a14b-2bb94fe44255 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:26.342316 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c466da6-8389-49da-819b-9f62e35d5bdd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:26.343057 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c466da6-8389-49da-819b-9f62e35d5bdd admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:26.343336 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c466da6-8389-49da-819b-9f62e35d5bdd admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:26.343608 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c466da6-8389-49da-819b-9f62e35d5bdd admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:26.344263 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c466da6-8389-49da-819b-9f62e35d5bdd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:26.345841 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c466da6-8389-49da-819b-9f62e35d5bdd admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:26.356964 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1329/2649] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:59:26 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:59:26.361345 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f2f2f66-14b7-4d6b-9a18-4c015ae7edeb admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:26.371902 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f2f2f66-14b7-4d6b-9a18-4c015ae7edeb admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:26.376835 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faad50bb-3b68-4836-a53a-e682dc185a00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:26.377455 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faad50bb-3b68-4836-a53a-e682dc185a00 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:26.377738 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faad50bb-3b68-4836-a53a-e682dc185a00 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:26.378118 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faad50bb-3b68-4836-a53a-e682dc185a00 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:26.378649 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faad50bb-3b68-4836-a53a-e682dc185a00 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:26.380303 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faad50bb-3b68-4836-a53a-e682dc185a00 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:26.391392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1321/2650] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:59:26 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 11 00:59:26.395324 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c466da6-8389-49da-819b-9f62e35d5bdd admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:26.405880 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c466da6-8389-49da-819b-9f62e35d5bdd admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:26.410735 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8e4c412-943e-4eee-88fb-42ed445de701 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:26.411415 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e4c412-943e-4eee-88fb-42ed445de701 admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:26.412259 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e4c412-943e-4eee-88fb-42ed445de701 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:26.412535 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e4c412-943e-4eee-88fb-42ed445de701 admin admin] PATH_INFO: `/v3/projects/2376212c577c4ecfa8f2294eeff906ae/users/aee78f3bd14c4922a298eaf81c4a20c1/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:26.426875 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8e4c412-943e-4eee-88fb-42ed445de701 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2376212c577c4ecfa8f2294eeff906ae, user_id=aee78f3bd14c4922a298eaf81c4a20c1, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:26.429579 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8e4c412-943e-4eee-88fb-42ed445de701 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:26.441789 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1330/2651] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:59:26 2022] PUT /identity/v3/projects/2376212c577c4ecfa8f2294eeff906ae/users/aee78f3bd14c4922a298eaf81c4a20c1/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:59:26.452587 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3029a2e8-7d0f-46a3-a098-71f46142dcb4 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:26.452902 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3029a2e8-7d0f-46a3-a098-71f46142dcb4 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:26.453172 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3029a2e8-7d0f-46a3-a098-71f46142dcb4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:26.480956 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-3029a2e8-7d0f-46a3-a098-71f46142dcb4 None None] MFA Rules not processed for user `aee78f3bd14c4922a298eaf81c4a20c1`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:59:26.482152 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-3029a2e8-7d0f-46a3-a098-71f46142dcb4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:59:26.530871 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:59:26.548505 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1322/2652] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:59:26 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:59:26.553425 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8e4c412-943e-4eee-88fb-42ed445de701 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:26.619901 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8e4c412-943e-4eee-88fb-42ed445de701 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:26.624543 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f30700f-6a97-4161-a525-e81c423337b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:26.625139 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f30700f-6a97-4161-a525-e81c423337b2 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:26.625415 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f30700f-6a97-4161-a525-e81c423337b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:26.625682 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f30700f-6a97-4161-a525-e81c423337b2 admin admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:26.626321 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f30700f-6a97-4161-a525-e81c423337b2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:26.628238 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f30700f-6a97-4161-a525-e81c423337b2 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:26.655145 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1331/2653] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:59:26 2022] POST /identity/v3/projects => generated 365 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:59:26.659444 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3029a2e8-7d0f-46a3-a098-71f46142dcb4 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:26.670418 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3029a2e8-7d0f-46a3-a098-71f46142dcb4 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:26.725851 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a9c838-8abc-4e62-8afe-fb58ee1da31e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:26.726456 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a9c838-8abc-4e62-8afe-fb58ee1da31e admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:26.726733 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a9c838-8abc-4e62-8afe-fb58ee1da31e admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:26.727033 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a9c838-8abc-4e62-8afe-fb58ee1da31e admin admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:26.727688 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03a9c838-8abc-4e62-8afe-fb58ee1da31e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:26.729600 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03a9c838-8abc-4e62-8afe-fb58ee1da31e admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:26.750358 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1323/2654] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:59:26 2022] POST /identity/v3/users => generated 289 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:59:26.754033 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f30700f-6a97-4161-a525-e81c423337b2 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:26.764843 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f30700f-6a97-4161-a525-e81c423337b2 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:26.769340 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e759d33-b516-4b99-b2af-4983b8a715b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:26.769927 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e759d33-b516-4b99-b2af-4983b8a715b7 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:26.770200 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e759d33-b516-4b99-b2af-4983b8a715b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:26.770492 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e759d33-b516-4b99-b2af-4983b8a715b7 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:26.771713 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e759d33-b516-4b99-b2af-4983b8a715b7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:26.773268 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e759d33-b516-4b99-b2af-4983b8a715b7 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:26.783716 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1332/2655] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:59:26 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:59:26.787785 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a9c838-8abc-4e62-8afe-fb58ee1da31e admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:26.797899 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a9c838-8abc-4e62-8afe-fb58ee1da31e admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:26.803079 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd1a675-6450-4c55-9509-66c5666b57a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:26.803825 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd1a675-6450-4c55-9509-66c5666b57a8 admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:26.804104 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd1a675-6450-4c55-9509-66c5666b57a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:26.804375 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd1a675-6450-4c55-9509-66c5666b57a8 admin admin] PATH_INFO: `/v3/projects/84711ed27fdc4e66a1925a19bc129961/users/1296a164c57646f5870c8596dfa2fe20/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:26.819478 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdd1a675-6450-4c55-9509-66c5666b57a8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=84711ed27fdc4e66a1925a19bc129961, user_id=1296a164c57646f5870c8596dfa2fe20, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:26.822132 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdd1a675-6450-4c55-9509-66c5666b57a8 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:26.835935 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1324/2656] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:59:26 2022] PUT /identity/v3/projects/84711ed27fdc4e66a1925a19bc129961/users/1296a164c57646f5870c8596dfa2fe20/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:59:26.839848 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e759d33-b516-4b99-b2af-4983b8a715b7 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:26.850469 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e759d33-b516-4b99-b2af-4983b8a715b7 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:26.859311 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:59:26.911465 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc512e1-a234-4299-b3a0-bae5993f2b2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:26.912052 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc512e1-a234-4299-b3a0-bae5993f2b2d admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:26.912330 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc512e1-a234-4299-b3a0-bae5993f2b2d admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:26.912613 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc512e1-a234-4299-b3a0-bae5993f2b2d admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:26.913283 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dc512e1-a234-4299-b3a0-bae5993f2b2d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:26.914915 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dc512e1-a234-4299-b3a0-bae5993f2b2d admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:26.925429 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1333/2657] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:59:26 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:59:26.930310 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd1a675-6450-4c55-9509-66c5666b57a8 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:26.941121 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd1a675-6450-4c55-9509-66c5666b57a8 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:26.946629 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e48ef383-1d55-4075-a12f-5f43e50c976d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:26.947345 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48ef383-1d55-4075-a12f-5f43e50c976d admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:26.947624 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48ef383-1d55-4075-a12f-5f43e50c976d admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:26.947896 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48ef383-1d55-4075-a12f-5f43e50c976d admin admin] PATH_INFO: `/v3/projects/84711ed27fdc4e66a1925a19bc129961/users/1296a164c57646f5870c8596dfa2fe20/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:26.949848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e48ef383-1d55-4075-a12f-5f43e50c976d admin admin] RBAC: Authorizing `identity:create_grant(project_id=84711ed27fdc4e66a1925a19bc129961, user_id=1296a164c57646f5870c8596dfa2fe20, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:26.952529 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e48ef383-1d55-4075-a12f-5f43e50c976d admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:26.963862 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1325/2658] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:59:26 2022] PUT /identity/v3/projects/84711ed27fdc4e66a1925a19bc129961/users/1296a164c57646f5870c8596dfa2fe20/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:59:26.974489 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a37d6bc-f912-441d-bfdf-d54c1dc85981 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:26.974778 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a37d6bc-f912-441d-bfdf-d54c1dc85981 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:26.975066 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a37d6bc-f912-441d-bfdf-d54c1dc85981 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:27.006150 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-2a37d6bc-f912-441d-bfdf-d54c1dc85981 None None] MFA Rules not processed for user `1296a164c57646f5870c8596dfa2fe20`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:59:27.007589 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-2a37d6bc-f912-441d-bfdf-d54c1dc85981 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:59:27.079606 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1334/2659] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:59:26 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:59:27.092276 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e48ef383-1d55-4075-a12f-5f43e50c976d admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:27.094332 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-e48ef383-1d55-4075-a12f-5f43e50c976d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:59:27.159559 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e48ef383-1d55-4075-a12f-5f43e50c976d admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:27.164024 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-111c7dbe-6098-4d9b-b8fd-6f7f6542da40 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:27.164626 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-111c7dbe-6098-4d9b-b8fd-6f7f6542da40 service cinder] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:27.164904 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-111c7dbe-6098-4d9b-b8fd-6f7f6542da40 service cinder] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:27.165170 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-111c7dbe-6098-4d9b-b8fd-6f7f6542da40 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:27.174508 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-111c7dbe-6098-4d9b-b8fd-6f7f6542da40 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:27.175979 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-111c7dbe-6098-4d9b-b8fd-6f7f6542da40 service cinder] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:27.186588 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1326/2660] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:59:27 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:59:27.263985 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a37d6bc-f912-441d-bfdf-d54c1dc85981 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:27.273482 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a37d6bc-f912-441d-bfdf-d54c1dc85981 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:27.277039 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2a70475-70b4-4884-a743-25c19a075735 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:27.277629 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a70475-70b4-4884-a743-25c19a075735 service cinder] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:27.277905 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a70475-70b4-4884-a743-25c19a075735 service cinder] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:27.278172 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a70475-70b4-4884-a743-25c19a075735 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:27.334677 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2a70475-70b4-4884-a743-25c19a075735 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:27.336164 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2a70475-70b4-4884-a743-25c19a075735 service cinder] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:27.345201 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1335/2661] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:59:27 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:59:37.182121 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=74,keystone:INSERT=5 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:59:37.341157 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=85 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:59:45.162591 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-111c7dbe-6098-4d9b-b8fd-6f7f6542da40 service cinder] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:45.225313 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-111c7dbe-6098-4d9b-b8fd-6f7f6542da40 service cinder] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:45.230016 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01917321-efbe-496d-85cf-b7a790435b70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:45.230652 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01917321-efbe-496d-85cf-b7a790435b70 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:45.230973 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01917321-efbe-496d-85cf-b7a790435b70 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:45.231264 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01917321-efbe-496d-85cf-b7a790435b70 admin admin] PATH_INFO: `/v3/users/aee78f3bd14c4922a298eaf81c4a20c1` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:45.232515 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01917321-efbe-496d-85cf-b7a790435b70 admin admin] RBAC: Authorizing `identity:delete_user(user_id=aee78f3bd14c4922a298eaf81c4a20c1)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:45.234438 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01917321-efbe-496d-85cf-b7a790435b70 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:45.281755 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-01917321-efbe-496d-85cf-b7a790435b70 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'aee78f3bd14c4922a298eaf81c4a20c1'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:45.282401 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-01917321-efbe-496d-85cf-b7a790435b70 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'aee78f3bd14c4922a298eaf81c4a20c1'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:45.283032 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-01917321-efbe-496d-85cf-b7a790435b70 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'aee78f3bd14c4922a298eaf81c4a20c1'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:45.291501 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-01917321-efbe-496d-85cf-b7a790435b70 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'aee78f3bd14c4922a298eaf81c4a20c1'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:45.304651 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-01917321-efbe-496d-85cf-b7a790435b70 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'aee78f3bd14c4922a298eaf81c4a20c1'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:45.319105 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1327/2662] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:59:45 2022] DELETE /identity/v3/users/aee78f3bd14c4922a298eaf81c4a20c1 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:59:45.323548 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa66a154-42a9-45b9-bdd7-0b528bd27f16 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:45.324077 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa66a154-42a9-45b9-bdd7-0b528bd27f16 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:45.324201 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa66a154-42a9-45b9-bdd7-0b528bd27f16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:45.343134 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-aa66a154-42a9-45b9-bdd7-0b528bd27f16 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:59:45.344340 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-aa66a154-42a9-45b9-bdd7-0b528bd27f16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:59:45.403068 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1336/2663] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:59:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:59:45.413686 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01917321-efbe-496d-85cf-b7a790435b70 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:45.415091 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-01917321-efbe-496d-85cf-b7a790435b70 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:59:45.475153 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01917321-efbe-496d-85cf-b7a790435b70 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:45.479943 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a3c5d5-b7c7-4115-aaa6-b01165d09884 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:45.480556 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a3c5d5-b7c7-4115-aaa6-b01165d09884 service neutron] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:45.480830 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a3c5d5-b7c7-4115-aaa6-b01165d09884 service neutron] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:45.481096 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a3c5d5-b7c7-4115-aaa6-b01165d09884 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:45.491385 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a3c5d5-b7c7-4115-aaa6-b01165d09884 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:45.492895 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a3c5d5-b7c7-4115-aaa6-b01165d09884 service neutron] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:45.502635 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1328/2664] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:59:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:59:45.805386 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa66a154-42a9-45b9-bdd7-0b528bd27f16 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:45.806940 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-aa66a154-42a9-45b9-bdd7-0b528bd27f16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:59:45.820022 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa66a154-42a9-45b9-bdd7-0b528bd27f16 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:45.824849 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-690c8c1b-f82c-4588-97a4-be3d97e325d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:45.825701 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690c8c1b-f82c-4588-97a4-be3d97e325d5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:45.825978 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690c8c1b-f82c-4588-97a4-be3d97e325d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:45.826239 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690c8c1b-f82c-4588-97a4-be3d97e325d5 admin admin] PATH_INFO: `/v3/projects/2376212c577c4ecfa8f2294eeff906ae` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:45.827413 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-690c8c1b-f82c-4588-97a4-be3d97e325d5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2376212c577c4ecfa8f2294eeff906ae)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:45.830491 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-690c8c1b-f82c-4588-97a4-be3d97e325d5 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:45.879608 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-690c8c1b-f82c-4588-97a4-be3d97e325d5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2376212c577c4ecfa8f2294eeff906ae', 'request_id': 'req-690c8c1b-f82c-4588-97a4-be3d97e325d5'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:45.885946 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-690c8c1b-f82c-4588-97a4-be3d97e325d5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2376212c577c4ecfa8f2294eeff906ae', 'request_id': 'req-690c8c1b-f82c-4588-97a4-be3d97e325d5'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:45.893004 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-690c8c1b-f82c-4588-97a4-be3d97e325d5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2376212c577c4ecfa8f2294eeff906ae', 'request_id': 'req-690c8c1b-f82c-4588-97a4-be3d97e325d5'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:45.904169 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-690c8c1b-f82c-4588-97a4-be3d97e325d5 admin admin] The token cache is being invalidate because project 2376212c577c4ecfa8f2294eeff906ae was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87924) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:59:45.904486 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-690c8c1b-f82c-4588-97a4-be3d97e325d5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:45.905046 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-690c8c1b-f82c-4588-97a4-be3d97e325d5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:45.906909 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1337/2665] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:59:45 2022] DELETE /identity/v3/projects/2376212c577c4ecfa8f2294eeff906ae => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:59:45.911737 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a3c5d5-b7c7-4115-aaa6-b01165d09884 service neutron] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:45.921515 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a3c5d5-b7c7-4115-aaa6-b01165d09884 service neutron] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:45.940279 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:59:45.981517 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21bcf8a9-e2d2-4e93-89ce-03a9c7169a5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:45.982148 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21bcf8a9-e2d2-4e93-89ce-03a9c7169a5a admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:45.982427 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21bcf8a9-e2d2-4e93-89ce-03a9c7169a5a admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:45.982701 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21bcf8a9-e2d2-4e93-89ce-03a9c7169a5a admin admin] PATH_INFO: `/v3/users/1296a164c57646f5870c8596dfa2fe20` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:45.983881 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21bcf8a9-e2d2-4e93-89ce-03a9c7169a5a admin admin] RBAC: Authorizing `identity:delete_user(user_id=1296a164c57646f5870c8596dfa2fe20)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:45.985847 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21bcf8a9-e2d2-4e93-89ce-03a9c7169a5a admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:46.031695 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-21bcf8a9-e2d2-4e93-89ce-03a9c7169a5a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1296a164c57646f5870c8596dfa2fe20'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:46.032346 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-21bcf8a9-e2d2-4e93-89ce-03a9c7169a5a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1296a164c57646f5870c8596dfa2fe20'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:46.032951 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-21bcf8a9-e2d2-4e93-89ce-03a9c7169a5a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1296a164c57646f5870c8596dfa2fe20'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:46.039834 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-21bcf8a9-e2d2-4e93-89ce-03a9c7169a5a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1296a164c57646f5870c8596dfa2fe20'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:46.051450 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-21bcf8a9-e2d2-4e93-89ce-03a9c7169a5a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1296a164c57646f5870c8596dfa2fe20'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:46.063708 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1329/2666] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:59:45 2022] DELETE /identity/v3/users/1296a164c57646f5870c8596dfa2fe20 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:59:46.624377 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-690c8c1b-f82c-4588-97a4-be3d97e325d5 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:46.625882 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-690c8c1b-f82c-4588-97a4-be3d97e325d5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:59:46.639687 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-690c8c1b-f82c-4588-97a4-be3d97e325d5 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:46.709217 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1bddc00-8792-49f8-8d4c-70020ec4c9e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:46.709859 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bddc00-8792-49f8-8d4c-70020ec4c9e3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:46.710139 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bddc00-8792-49f8-8d4c-70020ec4c9e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:46.710409 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bddc00-8792-49f8-8d4c-70020ec4c9e3 admin admin] PATH_INFO: `/v3/projects/84711ed27fdc4e66a1925a19bc129961` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:46.711559 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1bddc00-8792-49f8-8d4c-70020ec4c9e3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=84711ed27fdc4e66a1925a19bc129961)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:46.713491 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1bddc00-8792-49f8-8d4c-70020ec4c9e3 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:46.755624 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:59:46.778492 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-b1bddc00-8792-49f8-8d4c-70020ec4c9e3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '84711ed27fdc4e66a1925a19bc129961', 'request_id': 'req-b1bddc00-8792-49f8-8d4c-70020ec4c9e3'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:46.805786 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-b1bddc00-8792-49f8-8d4c-70020ec4c9e3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '84711ed27fdc4e66a1925a19bc129961', 'request_id': 'req-b1bddc00-8792-49f8-8d4c-70020ec4c9e3'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:46.827703 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-b1bddc00-8792-49f8-8d4c-70020ec4c9e3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '84711ed27fdc4e66a1925a19bc129961', 'request_id': 'req-b1bddc00-8792-49f8-8d4c-70020ec4c9e3'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:46.854949 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-b1bddc00-8792-49f8-8d4c-70020ec4c9e3 admin admin] The token cache is being invalidate because project 84711ed27fdc4e66a1925a19bc129961 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87924) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:59:46.855319 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-b1bddc00-8792-49f8-8d4c-70020ec4c9e3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:46.856047 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-b1bddc00-8792-49f8-8d4c-70020ec4c9e3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:46.857929 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1338/2667] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:59:46 2022] DELETE /identity/v3/projects/84711ed27fdc4e66a1925a19bc129961 => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:59:46.863524 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4667d1a-96ba-4d41-bf72-2a49b90c9a27 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:46.865474 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4667d1a-96ba-4d41-bf72-2a49b90c9a27 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:46.865474 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4667d1a-96ba-4d41-bf72-2a49b90c9a27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:46.887603 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-a4667d1a-96ba-4d41-bf72-2a49b90c9a27 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:59:46.888853 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-a4667d1a-96ba-4d41-bf72-2a49b90c9a27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:59:46.961741 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1330/2668] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:59:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:59:46.964014 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0baad55-20bc-4e52-a330-1268cf026e48 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:46.964298 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0baad55-20bc-4e52-a330-1268cf026e48 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:46.964587 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0baad55-20bc-4e52-a330-1268cf026e48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:46.987148 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-f0baad55-20bc-4e52-a330-1268cf026e48 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:59:46.988372 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-f0baad55-20bc-4e52-a330-1268cf026e48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:59:46.997860 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1339/2669] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:59:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:59:47.002152 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4667d1a-96ba-4d41-bf72-2a49b90c9a27 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:47.014983 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4667d1a-96ba-4d41-bf72-2a49b90c9a27 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:47.020782 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f8b54b1-8dc9-4811-b63a-5d335adfae22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:47.021125 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8b54b1-8dc9-4811-b63a-5d335adfae22 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:47.021501 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8b54b1-8dc9-4811-b63a-5d335adfae22 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:47.021773 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8b54b1-8dc9-4811-b63a-5d335adfae22 admin admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:47.022544 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f8b54b1-8dc9-4811-b63a-5d335adfae22 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:59:47.022899 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f8b54b1-8dc9-4811-b63a-5d335adfae22 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:47.024605 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f8b54b1-8dc9-4811-b63a-5d335adfae22 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:47.039626 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1331/2670] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 00:59:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 00:59:47.044145 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0baad55-20bc-4e52-a330-1268cf026e48 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:47.054474 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0baad55-20bc-4e52-a330-1268cf026e48 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:47.060628 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb582ea9-7b57-4484-8204-43b92ab28fa3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:47.061283 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb582ea9-7b57-4484-8204-43b92ab28fa3 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:47.061559 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb582ea9-7b57-4484-8204-43b92ab28fa3 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:47.061988 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb582ea9-7b57-4484-8204-43b92ab28fa3 admin admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:47.062743 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb582ea9-7b57-4484-8204-43b92ab28fa3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:47.065389 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb582ea9-7b57-4484-8204-43b92ab28fa3 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:47.114605 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1340/2671] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:59:47 2022] POST /identity/v3/projects => generated 369 bytes in 71 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:59:47.119200 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f8b54b1-8dc9-4811-b63a-5d335adfae22 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:47.130229 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f8b54b1-8dc9-4811-b63a-5d335adfae22 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:47.162348 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:59:47.199266 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d67d2fc-3620-498a-8fb5-0853c6a3e9db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:47.200228 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d67d2fc-3620-498a-8fb5-0853c6a3e9db admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:47.200567 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d67d2fc-3620-498a-8fb5-0853c6a3e9db admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:47.200841 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d67d2fc-3620-498a-8fb5-0853c6a3e9db admin admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:47.201484 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d67d2fc-3620-498a-8fb5-0853c6a3e9db admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:47.203547 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d67d2fc-3620-498a-8fb5-0853c6a3e9db admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:47.229280 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1332/2672] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:59:47 2022] POST /identity/v3/users => generated 285 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:59:47.233049 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb582ea9-7b57-4484-8204-43b92ab28fa3 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:47.247171 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb582ea9-7b57-4484-8204-43b92ab28fa3 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:47.252424 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1584e92a-fc16-4391-88c5-9db2a2234f17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:47.253328 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1584e92a-fc16-4391-88c5-9db2a2234f17 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:47.253328 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1584e92a-fc16-4391-88c5-9db2a2234f17 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:47.253649 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1584e92a-fc16-4391-88c5-9db2a2234f17 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:47.254210 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1584e92a-fc16-4391-88c5-9db2a2234f17 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:47.259456 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1584e92a-fc16-4391-88c5-9db2a2234f17 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:47.272762 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1341/2673] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:59:47 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:59:47.276993 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d67d2fc-3620-498a-8fb5-0853c6a3e9db admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:47.289870 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d67d2fc-3620-498a-8fb5-0853c6a3e9db admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:47.295061 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15f108fb-591c-403d-bccd-14ca6be35522 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:47.295665 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f108fb-591c-403d-bccd-14ca6be35522 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:47.295944 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f108fb-591c-403d-bccd-14ca6be35522 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:47.296212 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f108fb-591c-403d-bccd-14ca6be35522 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:47.296858 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15f108fb-591c-403d-bccd-14ca6be35522 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:47.298601 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15f108fb-591c-403d-bccd-14ca6be35522 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:47.313550 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1333/2674] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:59:47 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:59:47.315504 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1584e92a-fc16-4391-88c5-9db2a2234f17 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:47.330836 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1584e92a-fc16-4391-88c5-9db2a2234f17 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:47.336016 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c14f722-3fcf-472a-86ce-88879ac78891 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:47.336677 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c14f722-3fcf-472a-86ce-88879ac78891 admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:47.336978 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c14f722-3fcf-472a-86ce-88879ac78891 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:47.337224 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c14f722-3fcf-472a-86ce-88879ac78891 admin admin] PATH_INFO: `/v3/projects/347c6e41338045d29f61629a86b3d8d6/users/75a1a04455aa49daa03c9fd6d158e6dd/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:47.354015 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15f108fb-591c-403d-bccd-14ca6be35522 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:47.356162 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-15f108fb-591c-403d-bccd-14ca6be35522 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:59:47.368026 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c14f722-3fcf-472a-86ce-88879ac78891 admin admin] RBAC: Authorizing `identity:create_grant(project_id=347c6e41338045d29f61629a86b3d8d6, user_id=75a1a04455aa49daa03c9fd6d158e6dd, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:47.373897 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c14f722-3fcf-472a-86ce-88879ac78891 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:47.390913 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1342/2675] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:59:47 2022] PUT /identity/v3/projects/347c6e41338045d29f61629a86b3d8d6/users/75a1a04455aa49daa03c9fd6d158e6dd/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 76 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:59:47.401225 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-486e752f-2a79-4ec9-a76f-e71f5bc3e381 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:47.401505 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-486e752f-2a79-4ec9-a76f-e71f5bc3e381 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:47.401788 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-486e752f-2a79-4ec9-a76f-e71f5bc3e381 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:47.434055 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-486e752f-2a79-4ec9-a76f-e71f5bc3e381 None None] MFA Rules not processed for user `75a1a04455aa49daa03c9fd6d158e6dd`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:59:47.435401 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-486e752f-2a79-4ec9-a76f-e71f5bc3e381 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:59:47.454937 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15f108fb-591c-403d-bccd-14ca6be35522 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:47.506282 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1343/2676] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:59:47 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:59:47.510705 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-486e752f-2a79-4ec9-a76f-e71f5bc3e381 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:47.512025 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-486e752f-2a79-4ec9-a76f-e71f5bc3e381 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:59:47.523893 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd363748-56d2-49df-8a02-ee207331830c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '835def0e6c6d4365a2767051509eb0d6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'ResellerAdmin', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:47.524413 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd363748-56d2-49df-8a02-ee207331830c service nova] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:47.524690 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd363748-56d2-49df-8a02-ee207331830c service nova] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:47.525014 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd363748-56d2-49df-8a02-ee207331830c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:47.527245 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-dd363748-56d2-49df-8a02-ee207331830c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:59:47.555013 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:DELETE=6,keystone:SELECT=91,keystone:INSERT=3 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:59:47.572258 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-486e752f-2a79-4ec9-a76f-e71f5bc3e381 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:47.576063 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:59:47.577760 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a79c5f3-390f-400d-a4aa-aafa85e31554 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:47.578331 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a79c5f3-390f-400d-a4aa-aafa85e31554 service cinder] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:47.578641 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a79c5f3-390f-400d-a4aa-aafa85e31554 service cinder] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:47.579064 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a79c5f3-390f-400d-a4aa-aafa85e31554 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:47.581760 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-2a79c5f3-390f-400d-a4aa-aafa85e31554 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:59:47.596762 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd363748-56d2-49df-8a02-ee207331830c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:47.598456 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd363748-56d2-49df-8a02-ee207331830c service nova] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:47.608607 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1334/2677] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:59:47 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 256 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 11 00:59:47.609295 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd363748-56d2-49df-8a02-ee207331830c service nova] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:47.651626 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a79c5f3-390f-400d-a4aa-aafa85e31554 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:47.653688 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a79c5f3-390f-400d-a4aa-aafa85e31554 service cinder] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:47.665493 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1344/2678] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:59:47 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 159 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:59:47.686067 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd363748-56d2-49df-8a02-ee207331830c service nova] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:47.697186 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d65dbf-b505-4a30-82cf-9b84c931ba39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:47.698149 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d65dbf-b505-4a30-82cf-9b84c931ba39 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:47.698742 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d65dbf-b505-4a30-82cf-9b84c931ba39 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:47.699374 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d65dbf-b505-4a30-82cf-9b84c931ba39 admin admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:47.700014 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0d65dbf-b505-4a30-82cf-9b84c931ba39 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:47.701977 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0d65dbf-b505-4a30-82cf-9b84c931ba39 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:47.733203 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1335/2679] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:59:47 2022] POST /identity/v3/projects => generated 367 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:59:47.737506 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a79c5f3-390f-400d-a4aa-aafa85e31554 service cinder] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:47.749447 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a79c5f3-390f-400d-a4aa-aafa85e31554 service cinder] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:47.808056 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e449de5-c9f3-4403-9c0e-946ad761aff0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:47.808659 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e449de5-c9f3-4403-9c0e-946ad761aff0 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:47.808930 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e449de5-c9f3-4403-9c0e-946ad761aff0 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:47.809195 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e449de5-c9f3-4403-9c0e-946ad761aff0 admin admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:47.809812 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e449de5-c9f3-4403-9c0e-946ad761aff0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:47.811748 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e449de5-c9f3-4403-9c0e-946ad761aff0 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:47.832300 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1345/2680] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:59:47 2022] POST /identity/v3/users => generated 290 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:59:47.836956 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d65dbf-b505-4a30-82cf-9b84c931ba39 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:47.847222 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d65dbf-b505-4a30-82cf-9b84c931ba39 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:47.852520 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76ca8589-24fe-4367-a00b-1f29aae6c40c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:47.853111 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ca8589-24fe-4367-a00b-1f29aae6c40c admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:47.853392 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ca8589-24fe-4367-a00b-1f29aae6c40c admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:47.853664 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ca8589-24fe-4367-a00b-1f29aae6c40c admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:47.854304 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76ca8589-24fe-4367-a00b-1f29aae6c40c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:47.855938 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76ca8589-24fe-4367-a00b-1f29aae6c40c admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:47.867300 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1336/2681] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:59:47 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:59:47.871690 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e449de5-c9f3-4403-9c0e-946ad761aff0 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:47.881555 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e449de5-c9f3-4403-9c0e-946ad761aff0 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:47.887739 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af3a21ce-d5a7-46b7-a815-a7e353bd26d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:47.888401 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af3a21ce-d5a7-46b7-a815-a7e353bd26d9 admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:47.888679 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af3a21ce-d5a7-46b7-a815-a7e353bd26d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:47.888953 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af3a21ce-d5a7-46b7-a815-a7e353bd26d9 admin admin] PATH_INFO: `/v3/projects/f2ad3d42aae8491c8648a1e233168437/users/7d7c4346cc80405599094988ba1a78f4/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:47.905860 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af3a21ce-d5a7-46b7-a815-a7e353bd26d9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f2ad3d42aae8491c8648a1e233168437, user_id=7d7c4346cc80405599094988ba1a78f4, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:47.908792 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af3a21ce-d5a7-46b7-a815-a7e353bd26d9 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:47.931900 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1346/2682] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:59:47 2022] PUT /identity/v3/projects/f2ad3d42aae8491c8648a1e233168437/users/7d7c4346cc80405599094988ba1a78f4/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:59:47.935727 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76ca8589-24fe-4367-a00b-1f29aae6c40c admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:47.947625 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76ca8589-24fe-4367-a00b-1f29aae6c40c admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:48.018303 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1977c1a4-e90f-4fa8-b1d8-1adc4740c66b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:48.018957 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1977c1a4-e90f-4fa8-b1d8-1adc4740c66b admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:48.019238 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1977c1a4-e90f-4fa8-b1d8-1adc4740c66b admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:48.019509 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1977c1a4-e90f-4fa8-b1d8-1adc4740c66b admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:48.020777 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1977c1a4-e90f-4fa8-b1d8-1adc4740c66b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:48.022950 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1977c1a4-e90f-4fa8-b1d8-1adc4740c66b admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:48.040516 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1337/2683] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:59:47 2022] GET /identity/v3/roles => generated 1420 bytes in 105 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:59:48.044745 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af3a21ce-d5a7-46b7-a815-a7e353bd26d9 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:48.057022 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af3a21ce-d5a7-46b7-a815-a7e353bd26d9 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:48.061748 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfc859d8-0f40-4444-972b-8f15fb36b33c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:48.062474 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc859d8-0f40-4444-972b-8f15fb36b33c admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:48.062752 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc859d8-0f40-4444-972b-8f15fb36b33c admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:48.063054 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc859d8-0f40-4444-972b-8f15fb36b33c admin admin] PATH_INFO: `/v3/projects/f2ad3d42aae8491c8648a1e233168437/users/7d7c4346cc80405599094988ba1a78f4/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:48.064974 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfc859d8-0f40-4444-972b-8f15fb36b33c admin admin] RBAC: Authorizing `identity:create_grant(project_id=f2ad3d42aae8491c8648a1e233168437, user_id=7d7c4346cc80405599094988ba1a78f4, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:48.067765 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfc859d8-0f40-4444-972b-8f15fb36b33c admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:48.080364 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1347/2684] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:59:48 2022] PUT /identity/v3/projects/f2ad3d42aae8491c8648a1e233168437/users/7d7c4346cc80405599094988ba1a78f4/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 00:59:48.092246 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cebbdfce-5b2a-4b78-aa27-49d67428f35d None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:48.092747 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cebbdfce-5b2a-4b78-aa27-49d67428f35d None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:48.093097 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cebbdfce-5b2a-4b78-aa27-49d67428f35d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:48.130559 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-cebbdfce-5b2a-4b78-aa27-49d67428f35d None None] MFA Rules not processed for user `7d7c4346cc80405599094988ba1a78f4`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:59:48.132692 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-cebbdfce-5b2a-4b78-aa27-49d67428f35d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:59:48.177780 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:59:48.208393 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1338/2685] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:59:48 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:59:48.219537 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfc859d8-0f40-4444-972b-8f15fb36b33c admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:48.221764 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-dfc859d8-0f40-4444-972b-8f15fb36b33c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:59:48.277286 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:59:48.284308 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfc859d8-0f40-4444-972b-8f15fb36b33c admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:48.289943 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95c70aa-8bec-4be8-98f6-4fc8b42da495 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:48.291085 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95c70aa-8bec-4be8-98f6-4fc8b42da495 service cinder] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:48.291464 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95c70aa-8bec-4be8-98f6-4fc8b42da495 service cinder] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:48.291896 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95c70aa-8bec-4be8-98f6-4fc8b42da495 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:48.302208 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a95c70aa-8bec-4be8-98f6-4fc8b42da495 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:48.303724 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a95c70aa-8bec-4be8-98f6-4fc8b42da495 service cinder] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:48.313963 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1348/2686] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:59:48 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:59:49.738910 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cebbdfce-5b2a-4b78-aa27-49d67428f35d None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:49.804843 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cebbdfce-5b2a-4b78-aa27-49d67428f35d None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:49.809401 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2876e3f-dc3c-4841-bd68-67417b9aeac9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:49.810033 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2876e3f-dc3c-4841-bd68-67417b9aeac9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:49.810311 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2876e3f-dc3c-4841-bd68-67417b9aeac9 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:49.810592 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2876e3f-dc3c-4841-bd68-67417b9aeac9 admin admin] PATH_INFO: `/v3/users/75a1a04455aa49daa03c9fd6d158e6dd` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:49.811746 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2876e3f-dc3c-4841-bd68-67417b9aeac9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=75a1a04455aa49daa03c9fd6d158e6dd)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:49.813661 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2876e3f-dc3c-4841-bd68-67417b9aeac9 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:49.864875 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-b2876e3f-dc3c-4841-bd68-67417b9aeac9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '75a1a04455aa49daa03c9fd6d158e6dd'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:49.865597 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-b2876e3f-dc3c-4841-bd68-67417b9aeac9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '75a1a04455aa49daa03c9fd6d158e6dd'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:49.866713 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-b2876e3f-dc3c-4841-bd68-67417b9aeac9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '75a1a04455aa49daa03c9fd6d158e6dd'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:49.874690 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-b2876e3f-dc3c-4841-bd68-67417b9aeac9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '75a1a04455aa49daa03c9fd6d158e6dd'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:49.888717 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-b2876e3f-dc3c-4841-bd68-67417b9aeac9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '75a1a04455aa49daa03c9fd6d158e6dd'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:49.911694 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1339/2687] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:59:49 2022] DELETE /identity/v3/users/75a1a04455aa49daa03c9fd6d158e6dd => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:59:49.913353 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250ab248-76c0-462f-a541-b44ac2482688 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:49.913842 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250ab248-76c0-462f-a541-b44ac2482688 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:49.914032 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250ab248-76c0-462f-a541-b44ac2482688 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:49.933888 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-250ab248-76c0-462f-a541-b44ac2482688 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:59:49.935164 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-250ab248-76c0-462f-a541-b44ac2482688 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:59:49.996402 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1349/2688] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:59:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:59:50.006846 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2876e3f-dc3c-4841-bd68-67417b9aeac9 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:50.008272 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-b2876e3f-dc3c-4841-bd68-67417b9aeac9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:59:50.073611 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2876e3f-dc3c-4841-bd68-67417b9aeac9 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:50.078237 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a647e141-181b-48cf-9afe-378e21a5d2b7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:50.079401 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a647e141-181b-48cf-9afe-378e21a5d2b7 service neutron] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:50.079695 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a647e141-181b-48cf-9afe-378e21a5d2b7 service neutron] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:50.079982 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a647e141-181b-48cf-9afe-378e21a5d2b7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:50.094604 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a647e141-181b-48cf-9afe-378e21a5d2b7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:50.096515 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a647e141-181b-48cf-9afe-378e21a5d2b7 service neutron] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:50.106064 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:59:50.115017 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1340/2689] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:59:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:59:50.139395 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250ab248-76c0-462f-a541-b44ac2482688 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:50.141566 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-250ab248-76c0-462f-a541-b44ac2482688 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:59:50.158495 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250ab248-76c0-462f-a541-b44ac2482688 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:50.162884 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a6145f-e9c6-4847-85b7-c7b55f3723ca service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:50.163277 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a6145f-e9c6-4847-85b7-c7b55f3723ca service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:50.163571 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a6145f-e9c6-4847-85b7-c7b55f3723ca service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:50.163831 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a6145f-e9c6-4847-85b7-c7b55f3723ca service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:50.165595 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-c2a6145f-e9c6-4847-85b7-c7b55f3723ca service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:59:50.182418 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2a6145f-e9c6-4847-85b7-c7b55f3723ca service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:50.184702 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2a6145f-e9c6-4847-85b7-c7b55f3723ca service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:50.199142 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1350/2690] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:59:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 60 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:59:50.411254 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a647e141-181b-48cf-9afe-378e21a5d2b7 service neutron] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:50.413053 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-a647e141-181b-48cf-9afe-378e21a5d2b7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:59:50.436134 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a647e141-181b-48cf-9afe-378e21a5d2b7 service neutron] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:50.449025 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e9976d-e388-444a-ac7c-a7f5e7709d6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:50.449846 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e9976d-e388-444a-ac7c-a7f5e7709d6b admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:50.450127 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e9976d-e388-444a-ac7c-a7f5e7709d6b admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:50.450398 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e9976d-e388-444a-ac7c-a7f5e7709d6b admin admin] PATH_INFO: `/v3/projects/347c6e41338045d29f61629a86b3d8d6` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:50.452339 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71e9976d-e388-444a-ac7c-a7f5e7709d6b admin admin] RBAC: Authorizing `identity:delete_project(project_id=347c6e41338045d29f61629a86b3d8d6)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:50.454292 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71e9976d-e388-444a-ac7c-a7f5e7709d6b admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:50.508679 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-71e9976d-e388-444a-ac7c-a7f5e7709d6b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '347c6e41338045d29f61629a86b3d8d6', 'request_id': 'req-71e9976d-e388-444a-ac7c-a7f5e7709d6b'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:50.517537 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-71e9976d-e388-444a-ac7c-a7f5e7709d6b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '347c6e41338045d29f61629a86b3d8d6', 'request_id': 'req-71e9976d-e388-444a-ac7c-a7f5e7709d6b'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:50.527795 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-71e9976d-e388-444a-ac7c-a7f5e7709d6b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '347c6e41338045d29f61629a86b3d8d6', 'request_id': 'req-71e9976d-e388-444a-ac7c-a7f5e7709d6b'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:50.547950 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-71e9976d-e388-444a-ac7c-a7f5e7709d6b admin admin] The token cache is being invalidate because project 347c6e41338045d29f61629a86b3d8d6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87923) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:59:50.548359 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-71e9976d-e388-444a-ac7c-a7f5e7709d6b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:50.549054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-71e9976d-e388-444a-ac7c-a7f5e7709d6b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:50.550582 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1341/2691] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:59:50 2022] DELETE /identity/v3/projects/347c6e41338045d29f61629a86b3d8d6 => generated 0 bytes in 140 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:59:50.555588 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a6145f-e9c6-4847-85b7-c7b55f3723ca service neutron] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:50.565761 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a6145f-e9c6-4847-85b7-c7b55f3723ca service neutron] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:50.631002 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ae868d-0e21-45ff-898e-f5e6a8134114 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:50.631562 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ae868d-0e21-45ff-898e-f5e6a8134114 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:50.631837 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ae868d-0e21-45ff-898e-f5e6a8134114 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:50.632106 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ae868d-0e21-45ff-898e-f5e6a8134114 admin admin] PATH_INFO: `/v3/users/7d7c4346cc80405599094988ba1a78f4` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:50.633408 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49ae868d-0e21-45ff-898e-f5e6a8134114 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7d7c4346cc80405599094988ba1a78f4)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:50.635387 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49ae868d-0e21-45ff-898e-f5e6a8134114 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:50.696702 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-49ae868d-0e21-45ff-898e-f5e6a8134114 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7d7c4346cc80405599094988ba1a78f4'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:50.697653 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-49ae868d-0e21-45ff-898e-f5e6a8134114 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7d7c4346cc80405599094988ba1a78f4'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:50.698432 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-49ae868d-0e21-45ff-898e-f5e6a8134114 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7d7c4346cc80405599094988ba1a78f4'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:50.705967 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-49ae868d-0e21-45ff-898e-f5e6a8134114 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7d7c4346cc80405599094988ba1a78f4'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:50.721867 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-49ae868d-0e21-45ff-898e-f5e6a8134114 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7d7c4346cc80405599094988ba1a78f4'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:50.726670 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:59:50.747180 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1351/2692] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:59:50 2022] DELETE /identity/v3/users/7d7c4346cc80405599094988ba1a78f4 => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:59:51.310029 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e9976d-e388-444a-ac7c-a7f5e7709d6b admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:51.311598 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-71e9976d-e388-444a-ac7c-a7f5e7709d6b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:59:51.333208 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e9976d-e388-444a-ac7c-a7f5e7709d6b admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:51.415831 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f94917eb-ee63-44be-bcbe-e5d4f39bf737 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:51.416485 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f94917eb-ee63-44be-bcbe-e5d4f39bf737 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:51.416765 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f94917eb-ee63-44be-bcbe-e5d4f39bf737 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:51.417036 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f94917eb-ee63-44be-bcbe-e5d4f39bf737 admin admin] PATH_INFO: `/v3/projects/f2ad3d42aae8491c8648a1e233168437` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:51.418387 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f94917eb-ee63-44be-bcbe-e5d4f39bf737 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f2ad3d42aae8491c8648a1e233168437)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:51.420369 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f94917eb-ee63-44be-bcbe-e5d4f39bf737 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:51.481070 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-f94917eb-ee63-44be-bcbe-e5d4f39bf737 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f2ad3d42aae8491c8648a1e233168437', 'request_id': 'req-f94917eb-ee63-44be-bcbe-e5d4f39bf737'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:51.489796 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-f94917eb-ee63-44be-bcbe-e5d4f39bf737 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f2ad3d42aae8491c8648a1e233168437', 'request_id': 'req-f94917eb-ee63-44be-bcbe-e5d4f39bf737'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:51.495858 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-f94917eb-ee63-44be-bcbe-e5d4f39bf737 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f2ad3d42aae8491c8648a1e233168437', 'request_id': 'req-f94917eb-ee63-44be-bcbe-e5d4f39bf737'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:51.512188 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-f94917eb-ee63-44be-bcbe-e5d4f39bf737 admin admin] The token cache is being invalidate because project f2ad3d42aae8491c8648a1e233168437 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87923) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:59:51.512475 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-f94917eb-ee63-44be-bcbe-e5d4f39bf737 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:51.513269 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-f94917eb-ee63-44be-bcbe-e5d4f39bf737 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:51.514891 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1342/2693] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:59:51 2022] DELETE /identity/v3/projects/f2ad3d42aae8491c8648a1e233168437 => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:59:51.520075 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6913a7-19a5-4ce4-8f62-c735c915df3b None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:51.521729 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6913a7-19a5-4ce4-8f62-c735c915df3b None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:51.522283 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6913a7-19a5-4ce4-8f62-c735c915df3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:51.568887 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-8a6913a7-19a5-4ce4-8f62-c735c915df3b None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:59:51.570174 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-8a6913a7-19a5-4ce4-8f62-c735c915df3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:59:51.639747 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1352/2694] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:59:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:59:51.648184 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21a1b42-4300-4433-bdfb-f1cb7c6e34c1 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:51.648475 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21a1b42-4300-4433-bdfb-f1cb7c6e34c1 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:51.648754 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21a1b42-4300-4433-bdfb-f1cb7c6e34c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:51.669557 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-e21a1b42-4300-4433-bdfb-f1cb7c6e34c1 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:59:51.670758 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-e21a1b42-4300-4433-bdfb-f1cb7c6e34c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:59:51.679229 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1343/2695] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:59:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:59:51.683525 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a6913a7-19a5-4ce4-8f62-c735c915df3b None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:51.694313 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a6913a7-19a5-4ce4-8f62-c735c915df3b None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:51.699309 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6696caf-1ef7-4812-b15e-9cf8b377bd92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:51.700726 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6696caf-1ef7-4812-b15e-9cf8b377bd92 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:51.700726 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6696caf-1ef7-4812-b15e-9cf8b377bd92 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:51.700726 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6696caf-1ef7-4812-b15e-9cf8b377bd92 admin admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:51.701111 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6696caf-1ef7-4812-b15e-9cf8b377bd92 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 00:59:51.701431 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6696caf-1ef7-4812-b15e-9cf8b377bd92 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:51.703097 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6696caf-1ef7-4812-b15e-9cf8b377bd92 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:51.715536 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1353/2696] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 00:59:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 11 00:59:51.719838 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21a1b42-4300-4433-bdfb-f1cb7c6e34c1 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:51.729460 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21a1b42-4300-4433-bdfb-f1cb7c6e34c1 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:51.734412 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f968995-b030-4a4b-b714-3625e4e9286c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:51.734852 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f968995-b030-4a4b-b714-3625e4e9286c admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:51.735463 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f968995-b030-4a4b-b714-3625e4e9286c admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:51.735463 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f968995-b030-4a4b-b714-3625e4e9286c admin admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:51.736053 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f968995-b030-4a4b-b714-3625e4e9286c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:51.738352 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f968995-b030-4a4b-b714-3625e4e9286c admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:51.748981 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:59:51.777191 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1344/2697] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:59:51 2022] POST /identity/v3/projects => generated 399 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:59:51.781567 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6696caf-1ef7-4812-b15e-9cf8b377bd92 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:51.792835 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6696caf-1ef7-4812-b15e-9cf8b377bd92 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:51.857691 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b916a7-d0fd-4f68-b6ff-a5941c2d6e10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:51.858519 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b916a7-d0fd-4f68-b6ff-a5941c2d6e10 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:51.858900 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b916a7-d0fd-4f68-b6ff-a5941c2d6e10 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:51.859240 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b916a7-d0fd-4f68-b6ff-a5941c2d6e10 admin admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:51.859891 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65b916a7-d0fd-4f68-b6ff-a5941c2d6e10 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:51.861799 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65b916a7-d0fd-4f68-b6ff-a5941c2d6e10 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:51.893573 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1354/2698] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:59:51 2022] POST /identity/v3/users => generated 300 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:59:51.897925 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f968995-b030-4a4b-b714-3625e4e9286c admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:51.909596 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f968995-b030-4a4b-b714-3625e4e9286c admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:51.915831 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cde3e6b3-f8c1-4599-9c70-43808e90f952 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:51.917295 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde3e6b3-f8c1-4599-9c70-43808e90f952 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:51.917672 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde3e6b3-f8c1-4599-9c70-43808e90f952 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:51.918042 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde3e6b3-f8c1-4599-9c70-43808e90f952 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:51.918998 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cde3e6b3-f8c1-4599-9c70-43808e90f952 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:51.921048 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cde3e6b3-f8c1-4599-9c70-43808e90f952 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:51.936693 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1345/2699] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:59:51 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:59:51.940403 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b916a7-d0fd-4f68-b6ff-a5941c2d6e10 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:51.951779 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b916a7-d0fd-4f68-b6ff-a5941c2d6e10 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:51.956693 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-968ce356-4eeb-4e48-b552-ff351e865c18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:51.957389 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968ce356-4eeb-4e48-b552-ff351e865c18 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:51.957831 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968ce356-4eeb-4e48-b552-ff351e865c18 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:51.958124 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968ce356-4eeb-4e48-b552-ff351e865c18 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:51.958807 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-968ce356-4eeb-4e48-b552-ff351e865c18 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:51.960482 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-968ce356-4eeb-4e48-b552-ff351e865c18 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:51.972651 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1355/2700] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:59:51 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:59:51.977962 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cde3e6b3-f8c1-4599-9c70-43808e90f952 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:51.990029 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cde3e6b3-f8c1-4599-9c70-43808e90f952 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:51.995186 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39770182-436a-4fcf-aa78-753300470eb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:51.995898 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39770182-436a-4fcf-aa78-753300470eb7 admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:51.996338 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39770182-436a-4fcf-aa78-753300470eb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:51.996621 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39770182-436a-4fcf-aa78-753300470eb7 admin admin] PATH_INFO: `/v3/projects/b877a93900ed48788d8cc8c0e3b88211/users/c3871492956c4a84a7263ca8551af809/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:52.014168 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39770182-436a-4fcf-aa78-753300470eb7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b877a93900ed48788d8cc8c0e3b88211, user_id=c3871492956c4a84a7263ca8551af809, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:52.016855 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39770182-436a-4fcf-aa78-753300470eb7 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:52.033051 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1346/2701] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:59:51 2022] PUT /identity/v3/projects/b877a93900ed48788d8cc8c0e3b88211/users/c3871492956c4a84a7263ca8551af809/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:59:52.046163 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-826454f1-7dfc-4a50-a348-dd12171a764a None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:52.046604 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-826454f1-7dfc-4a50-a348-dd12171a764a None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:52.047539 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-826454f1-7dfc-4a50-a348-dd12171a764a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:52.084789 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-826454f1-7dfc-4a50-a348-dd12171a764a None None] MFA Rules not processed for user `c3871492956c4a84a7263ca8551af809`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:59:52.086076 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-826454f1-7dfc-4a50-a348-dd12171a764a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:59:52.148402 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:59:52.169746 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1356/2702] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:59:52 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 00:59:52.174473 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39770182-436a-4fcf-aa78-753300470eb7 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:52.283043 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39770182-436a-4fcf-aa78-753300470eb7 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:52.287909 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10bac89f-bd35-40db-94d9-335cee858000 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:52.288529 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10bac89f-bd35-40db-94d9-335cee858000 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:52.288878 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10bac89f-bd35-40db-94d9-335cee858000 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:52.289498 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10bac89f-bd35-40db-94d9-335cee858000 admin admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:52.289829 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10bac89f-bd35-40db-94d9-335cee858000 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:52.292005 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10bac89f-bd35-40db-94d9-335cee858000 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:52.332170 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1347/2703] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 00:59:52 2022] POST /identity/v3/projects => generated 401 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 00:59:52.337465 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-826454f1-7dfc-4a50-a348-dd12171a764a None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:52.349988 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-826454f1-7dfc-4a50-a348-dd12171a764a None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:52.430392 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5093e442-3792-4c84-8d4c-3907587d4178 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:52.431120 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5093e442-3792-4c84-8d4c-3907587d4178 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:52.431408 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5093e442-3792-4c84-8d4c-3907587d4178 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:52.431677 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5093e442-3792-4c84-8d4c-3907587d4178 admin admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:52.432520 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5093e442-3792-4c84-8d4c-3907587d4178 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:52.434720 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5093e442-3792-4c84-8d4c-3907587d4178 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:52.466630 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1357/2704] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 00:59:52 2022] POST /identity/v3/users => generated 307 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 00:59:52.471263 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10bac89f-bd35-40db-94d9-335cee858000 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:52.489359 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10bac89f-bd35-40db-94d9-335cee858000 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:52.500511 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45fe7f55-bf02-48af-b30f-0483706bc9ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:52.501849 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45fe7f55-bf02-48af-b30f-0483706bc9ae admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:52.502232 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45fe7f55-bf02-48af-b30f-0483706bc9ae admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:52.502607 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45fe7f55-bf02-48af-b30f-0483706bc9ae admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:52.504730 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45fe7f55-bf02-48af-b30f-0483706bc9ae admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:52.506740 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45fe7f55-bf02-48af-b30f-0483706bc9ae admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:52.520354 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1348/2705] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:59:52 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:59:52.525429 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5093e442-3792-4c84-8d4c-3907587d4178 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:52.539792 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5093e442-3792-4c84-8d4c-3907587d4178 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:52.548559 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8434ffc-6b76-4be8-8757-fa6e95cb02f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:52.550319 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8434ffc-6b76-4be8-8757-fa6e95cb02f1 admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:52.550773 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8434ffc-6b76-4be8-8757-fa6e95cb02f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:52.551855 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8434ffc-6b76-4be8-8757-fa6e95cb02f1 admin admin] PATH_INFO: `/v3/projects/7d515b6e5817481fad25a37e38170dfa/users/4dee88f12ef84a328565b349a0bc8a49/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:52.576082 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8434ffc-6b76-4be8-8757-fa6e95cb02f1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7d515b6e5817481fad25a37e38170dfa, user_id=4dee88f12ef84a328565b349a0bc8a49, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:52.579700 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8434ffc-6b76-4be8-8757-fa6e95cb02f1 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:52.608892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1358/2706] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:59:52 2022] PUT /identity/v3/projects/7d515b6e5817481fad25a37e38170dfa/users/4dee88f12ef84a328565b349a0bc8a49/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 81 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:59:52.611154 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45fe7f55-bf02-48af-b30f-0483706bc9ae admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:52.624509 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45fe7f55-bf02-48af-b30f-0483706bc9ae admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:52.696737 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52311619-8a33-4439-b2f2-f3eac94b24cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:52.698088 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52311619-8a33-4439-b2f2-f3eac94b24cc admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:52.698088 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52311619-8a33-4439-b2f2-f3eac94b24cc admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:52.698088 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52311619-8a33-4439-b2f2-f3eac94b24cc admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:52.698953 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52311619-8a33-4439-b2f2-f3eac94b24cc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:52.700765 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52311619-8a33-4439-b2f2-f3eac94b24cc admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:52.713532 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1349/2707] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 00:59:52 2022] GET /identity/v3/roles => generated 1420 bytes in 103 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 00:59:52.718206 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8434ffc-6b76-4be8-8757-fa6e95cb02f1 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:52.731891 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8434ffc-6b76-4be8-8757-fa6e95cb02f1 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:52.740529 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370c6b91-8b82-4b1a-9e44-d1fa42880fd0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:52.741269 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370c6b91-8b82-4b1a-9e44-d1fa42880fd0 admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:52.741884 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370c6b91-8b82-4b1a-9e44-d1fa42880fd0 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:52.741884 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370c6b91-8b82-4b1a-9e44-d1fa42880fd0 admin admin] PATH_INFO: `/v3/projects/7d515b6e5817481fad25a37e38170dfa/users/4dee88f12ef84a328565b349a0bc8a49/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:52.745151 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-370c6b91-8b82-4b1a-9e44-d1fa42880fd0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7d515b6e5817481fad25a37e38170dfa, user_id=4dee88f12ef84a328565b349a0bc8a49, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:52.748606 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-370c6b91-8b82-4b1a-9e44-d1fa42880fd0 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:52.761601 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1359/2708] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 00:59:52 2022] PUT /identity/v3/projects/7d515b6e5817481fad25a37e38170dfa/users/4dee88f12ef84a328565b349a0bc8a49/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 44 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:59:52.774049 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc65b34-f3cf-42dd-8fa8-a99133a0f33a None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:52.774447 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc65b34-f3cf-42dd-8fa8-a99133a0f33a None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:52.774818 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc65b34-f3cf-42dd-8fa8-a99133a0f33a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:52.813175 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-fdc65b34-f3cf-42dd-8fa8-a99133a0f33a None None] MFA Rules not processed for user `4dee88f12ef84a328565b349a0bc8a49`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:59:52.819459 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-fdc65b34-f3cf-42dd-8fa8-a99133a0f33a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:59:52.830930 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:59:52.929498 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1350/2709] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:59:52 2022] POST /identity/v3/auth/tokens => generated 3345 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:59:52.942671 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370c6b91-8b82-4b1a-9e44-d1fa42880fd0 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:52.945485 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-370c6b91-8b82-4b1a-9e44-d1fa42880fd0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:59:53.048352 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370c6b91-8b82-4b1a-9e44-d1fa42880fd0 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:53.052947 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6d6597-993e-48dd-8fb4-2ffc015e5f8f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:53.053569 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6d6597-993e-48dd-8fb4-2ffc015e5f8f service cinder] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:53.053848 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6d6597-993e-48dd-8fb4-2ffc015e5f8f service cinder] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:53.054120 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6d6597-993e-48dd-8fb4-2ffc015e5f8f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:53.067278 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc6d6597-993e-48dd-8fb4-2ffc015e5f8f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:53.067278 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc6d6597-993e-48dd-8fb4-2ffc015e5f8f service cinder] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:53.088219 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1360/2710] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:59:52 2022] GET /identity/v3/auth/tokens => generated 3345 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:59:53.179388 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc65b34-f3cf-42dd-8fa8-a99133a0f33a None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:53.190462 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc65b34-f3cf-42dd-8fa8-a99133a0f33a None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:53.195422 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9eac359-900c-41c6-bfb0-020dd3b23e7d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:53.196294 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9eac359-900c-41c6-bfb0-020dd3b23e7d service cinder] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:53.196717 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9eac359-900c-41c6-bfb0-020dd3b23e7d service cinder] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:53.196962 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9eac359-900c-41c6-bfb0-020dd3b23e7d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:53.284251 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9eac359-900c-41c6-bfb0-020dd3b23e7d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:53.286842 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9eac359-900c-41c6-bfb0-020dd3b23e7d service cinder] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:53.296709 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1351/2711] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:59:53 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 11 00:59:53.977308 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6d6597-993e-48dd-8fb4-2ffc015e5f8f service cinder] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:53.979092 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-fc6d6597-993e-48dd-8fb4-2ffc015e5f8f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:59:54.035053 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:59:54.061357 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6d6597-993e-48dd-8fb4-2ffc015e5f8f service cinder] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:54.070350 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b7dd80-674e-4ec4-886a-e08db43b85a0 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cdbfc2ccd73e4a5eb8c999be5b094619', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:54.071017 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b7dd80-674e-4ec4-886a-e08db43b85a0 service placement] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:54.071292 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b7dd80-674e-4ec4-886a-e08db43b85a0 service placement] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:54.071557 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b7dd80-674e-4ec4-886a-e08db43b85a0 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:54.073343 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-76b7dd80-674e-4ec4-886a-e08db43b85a0 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:59:54.086508 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76b7dd80-674e-4ec4-886a-e08db43b85a0 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:54.088171 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76b7dd80-674e-4ec4-886a-e08db43b85a0 service placement] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:54.100841 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1361/2712] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:59:53 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 122 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:59:57.362079 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9eac359-900c-41c6-bfb0-020dd3b23e7d service cinder] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:57.363992 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-f9eac359-900c-41c6-bfb0-020dd3b23e7d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:59:57.437142 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9eac359-900c-41c6-bfb0-020dd3b23e7d service cinder] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:57.442259 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c970ca-1487-4b2b-a90f-bc956ad2b794 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:57.442903 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c970ca-1487-4b2b-a90f-bc956ad2b794 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:57.443204 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c970ca-1487-4b2b-a90f-bc956ad2b794 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:57.443475 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c970ca-1487-4b2b-a90f-bc956ad2b794 admin admin] PATH_INFO: `/v3/users/967e0ef4e6364804a58d0d59c22cd6a8` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:57.444620 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36c970ca-1487-4b2b-a90f-bc956ad2b794 admin admin] RBAC: Authorizing `identity:delete_user(user_id=967e0ef4e6364804a58d0d59c22cd6a8)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:57.446565 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36c970ca-1487-4b2b-a90f-bc956ad2b794 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:57.516415 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:59:57.519180 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-36c970ca-1487-4b2b-a90f-bc956ad2b794 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '967e0ef4e6364804a58d0d59c22cd6a8'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:57.521390 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-36c970ca-1487-4b2b-a90f-bc956ad2b794 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '967e0ef4e6364804a58d0d59c22cd6a8'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:57.523254 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-36c970ca-1487-4b2b-a90f-bc956ad2b794 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '967e0ef4e6364804a58d0d59c22cd6a8'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:57.533128 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-36c970ca-1487-4b2b-a90f-bc956ad2b794 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '967e0ef4e6364804a58d0d59c22cd6a8'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:57.545962 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-36c970ca-1487-4b2b-a90f-bc956ad2b794 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '967e0ef4e6364804a58d0d59c22cd6a8'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:57.561169 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1352/2713] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:59:57 2022] DELETE /identity/v3/users/967e0ef4e6364804a58d0d59c22cd6a8 => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:59:57.743613 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b7dd80-674e-4ec4-886a-e08db43b85a0 service placement] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:57.762482 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b7dd80-674e-4ec4-886a-e08db43b85a0 service placement] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:57.820098 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c2b6486-97c2-46c9-ad4e-76a15cc42f43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:57.820759 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c2b6486-97c2-46c9-ad4e-76a15cc42f43 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:57.821039 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c2b6486-97c2-46c9-ad4e-76a15cc42f43 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:57.821311 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c2b6486-97c2-46c9-ad4e-76a15cc42f43 admin admin] PATH_INFO: `/v3/projects/d55d41063ec34fb6921c367025fa9af9` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:57.822458 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c2b6486-97c2-46c9-ad4e-76a15cc42f43 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d55d41063ec34fb6921c367025fa9af9)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:57.824416 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c2b6486-97c2-46c9-ad4e-76a15cc42f43 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:57.872167 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-1c2b6486-97c2-46c9-ad4e-76a15cc42f43 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd55d41063ec34fb6921c367025fa9af9', 'request_id': 'req-1c2b6486-97c2-46c9-ad4e-76a15cc42f43'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:57.879469 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-1c2b6486-97c2-46c9-ad4e-76a15cc42f43 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd55d41063ec34fb6921c367025fa9af9', 'request_id': 'req-1c2b6486-97c2-46c9-ad4e-76a15cc42f43'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:57.887427 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-1c2b6486-97c2-46c9-ad4e-76a15cc42f43 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd55d41063ec34fb6921c367025fa9af9', 'request_id': 'req-1c2b6486-97c2-46c9-ad4e-76a15cc42f43'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:57.899396 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-1c2b6486-97c2-46c9-ad4e-76a15cc42f43 admin admin] The token cache is being invalidate because project d55d41063ec34fb6921c367025fa9af9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87924) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:59:57.899821 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-1c2b6486-97c2-46c9-ad4e-76a15cc42f43 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:57.900677 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-1c2b6486-97c2-46c9-ad4e-76a15cc42f43 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:57.902423 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1362/2714] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:59:57 2022] DELETE /identity/v3/projects/d55d41063ec34fb6921c367025fa9af9 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:59:57.907111 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c970ca-1487-4b2b-a90f-bc956ad2b794 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:57.908588 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-36c970ca-1487-4b2b-a90f-bc956ad2b794 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:59:57.922270 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c970ca-1487-4b2b-a90f-bc956ad2b794 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:57.980353 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa05da4-be93-4231-92aa-85729dc95c1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:57.980978 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa05da4-be93-4231-92aa-85729dc95c1f admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:57.981252 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa05da4-be93-4231-92aa-85729dc95c1f admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:57.981520 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa05da4-be93-4231-92aa-85729dc95c1f admin admin] PATH_INFO: `/v3/users/96a1830ec63b4158a2ad31bdc84f8f3c` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:57.982637 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aa05da4-be93-4231-92aa-85729dc95c1f admin admin] RBAC: Authorizing `identity:delete_user(user_id=96a1830ec63b4158a2ad31bdc84f8f3c)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:57.984580 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aa05da4-be93-4231-92aa-85729dc95c1f admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:58.030379 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-9aa05da4-be93-4231-92aa-85729dc95c1f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '96a1830ec63b4158a2ad31bdc84f8f3c'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:58.031004 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-9aa05da4-be93-4231-92aa-85729dc95c1f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '96a1830ec63b4158a2ad31bdc84f8f3c'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:58.031881 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-9aa05da4-be93-4231-92aa-85729dc95c1f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '96a1830ec63b4158a2ad31bdc84f8f3c'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:58.039819 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-9aa05da4-be93-4231-92aa-85729dc95c1f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '96a1830ec63b4158a2ad31bdc84f8f3c'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:58.052552 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-9aa05da4-be93-4231-92aa-85729dc95c1f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '96a1830ec63b4158a2ad31bdc84f8f3c'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:58.065035 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1353/2715] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:59:57 2022] DELETE /identity/v3/users/96a1830ec63b4158a2ad31bdc84f8f3c => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:59:58.239760 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c2b6486-97c2-46c9-ad4e-76a15cc42f43 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:58.241197 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-1c2b6486-97c2-46c9-ad4e-76a15cc42f43 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:59:58.254716 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c2b6486-97c2-46c9-ad4e-76a15cc42f43 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:58.315121 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27df2636-0bb8-4fcf-99cf-0a0c01625f11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:58.315751 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27df2636-0bb8-4fcf-99cf-0a0c01625f11 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:58.316025 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27df2636-0bb8-4fcf-99cf-0a0c01625f11 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:58.316316 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27df2636-0bb8-4fcf-99cf-0a0c01625f11 admin admin] PATH_INFO: `/v3/projects/c0a6200260914ce48590b57b10f75f24` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:58.317632 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27df2636-0bb8-4fcf-99cf-0a0c01625f11 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c0a6200260914ce48590b57b10f75f24)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:58.319810 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27df2636-0bb8-4fcf-99cf-0a0c01625f11 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:58.329636 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:59:58.375800 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-27df2636-0bb8-4fcf-99cf-0a0c01625f11 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c0a6200260914ce48590b57b10f75f24', 'request_id': 'req-27df2636-0bb8-4fcf-99cf-0a0c01625f11'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:58.381893 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-27df2636-0bb8-4fcf-99cf-0a0c01625f11 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c0a6200260914ce48590b57b10f75f24', 'request_id': 'req-27df2636-0bb8-4fcf-99cf-0a0c01625f11'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:58.389380 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-27df2636-0bb8-4fcf-99cf-0a0c01625f11 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c0a6200260914ce48590b57b10f75f24', 'request_id': 'req-27df2636-0bb8-4fcf-99cf-0a0c01625f11'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:58.400387 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-27df2636-0bb8-4fcf-99cf-0a0c01625f11 admin admin] The token cache is being invalidate because project c0a6200260914ce48590b57b10f75f24 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87924) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:59:58.400668 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-27df2636-0bb8-4fcf-99cf-0a0c01625f11 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:58.401290 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-27df2636-0bb8-4fcf-99cf-0a0c01625f11 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:58.402760 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1363/2716] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:59:58 2022] DELETE /identity/v3/projects/c0a6200260914ce48590b57b10f75f24 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:59:58.705783 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa05da4-be93-4231-92aa-85729dc95c1f admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:58.708157 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-9aa05da4-be93-4231-92aa-85729dc95c1f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:59:58.720515 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa05da4-be93-4231-92aa-85729dc95c1f admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:58.763321 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:59:58.786338 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78cd3f45-897d-4df4-97be-c4be1768561b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:58.786926 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78cd3f45-897d-4df4-97be-c4be1768561b admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:58.787204 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78cd3f45-897d-4df4-97be-c4be1768561b admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:58.787475 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78cd3f45-897d-4df4-97be-c4be1768561b admin admin] PATH_INFO: `/v3/users/c3871492956c4a84a7263ca8551af809` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:58.788619 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78cd3f45-897d-4df4-97be-c4be1768561b admin admin] RBAC: Authorizing `identity:delete_user(user_id=c3871492956c4a84a7263ca8551af809)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:58.790746 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78cd3f45-897d-4df4-97be-c4be1768561b admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:58.839571 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-78cd3f45-897d-4df4-97be-c4be1768561b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c3871492956c4a84a7263ca8551af809'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:58.840258 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-78cd3f45-897d-4df4-97be-c4be1768561b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c3871492956c4a84a7263ca8551af809'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:58.840858 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-78cd3f45-897d-4df4-97be-c4be1768561b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c3871492956c4a84a7263ca8551af809'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:58.848111 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-78cd3f45-897d-4df4-97be-c4be1768561b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c3871492956c4a84a7263ca8551af809'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:58.863672 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-78cd3f45-897d-4df4-97be-c4be1768561b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c3871492956c4a84a7263ca8551af809'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:58.886876 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1354/2717] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:59:58 2022] DELETE /identity/v3/users/c3871492956c4a84a7263ca8551af809 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:59:58.891280 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3663b3ad-322e-4aa1-9458-e7b26c6a96c4 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:58.891697 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3663b3ad-322e-4aa1-9458-e7b26c6a96c4 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:58.892042 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3663b3ad-322e-4aa1-9458-e7b26c6a96c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:58.913955 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-3663b3ad-322e-4aa1-9458-e7b26c6a96c4 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 00:59:58.915194 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-3663b3ad-322e-4aa1-9458-e7b26c6a96c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:59:58.974805 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1364/2718] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 00:59:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 00:59:58.984612 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78cd3f45-897d-4df4-97be-c4be1768561b admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:58.986031 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-78cd3f45-897d-4df4-97be-c4be1768561b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:59:59.042624 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78cd3f45-897d-4df4-97be-c4be1768561b admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:59.046279 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27fe81de-7e99-43bd-b130-c699d14180f1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:59.046897 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fe81de-7e99-43bd-b130-c699d14180f1 service neutron] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:59.047174 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fe81de-7e99-43bd-b130-c699d14180f1 service neutron] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:59.047439 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fe81de-7e99-43bd-b130-c699d14180f1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:59.057000 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27fe81de-7e99-43bd-b130-c699d14180f1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:59.058414 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27fe81de-7e99-43bd-b130-c699d14180f1 service neutron] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:59.068238 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1355/2719] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 00:59:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 00:59:59.332690 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3663b3ad-322e-4aa1-9458-e7b26c6a96c4 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:59.334264 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-3663b3ad-322e-4aa1-9458-e7b26c6a96c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:59:59.346664 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3663b3ad-322e-4aa1-9458-e7b26c6a96c4 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:59.351527 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef3b6a45-aa97-4305-895a-76f3f397485b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:59.352156 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3b6a45-aa97-4305-895a-76f3f397485b admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:59.352431 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3b6a45-aa97-4305-895a-76f3f397485b admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:59.352700 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3b6a45-aa97-4305-895a-76f3f397485b admin admin] PATH_INFO: `/v3/projects/b877a93900ed48788d8cc8c0e3b88211` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:59.353826 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef3b6a45-aa97-4305-895a-76f3f397485b admin admin] RBAC: Authorizing `identity:delete_project(project_id=b877a93900ed48788d8cc8c0e3b88211)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:59.355787 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef3b6a45-aa97-4305-895a-76f3f397485b admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:59.403805 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-ef3b6a45-aa97-4305-895a-76f3f397485b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b877a93900ed48788d8cc8c0e3b88211', 'request_id': 'req-ef3b6a45-aa97-4305-895a-76f3f397485b'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:59.410603 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-ef3b6a45-aa97-4305-895a-76f3f397485b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b877a93900ed48788d8cc8c0e3b88211', 'request_id': 'req-ef3b6a45-aa97-4305-895a-76f3f397485b'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:59.418381 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-ef3b6a45-aa97-4305-895a-76f3f397485b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b877a93900ed48788d8cc8c0e3b88211', 'request_id': 'req-ef3b6a45-aa97-4305-895a-76f3f397485b'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:59.429429 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-ef3b6a45-aa97-4305-895a-76f3f397485b admin admin] The token cache is being invalidate because project b877a93900ed48788d8cc8c0e3b88211 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87924) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 00:59:59.429716 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-ef3b6a45-aa97-4305-895a-76f3f397485b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:59.430344 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-ef3b6a45-aa97-4305-895a-76f3f397485b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:59.431931 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1365/2720] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:59:59 2022] DELETE /identity/v3/projects/b877a93900ed48788d8cc8c0e3b88211 => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:59:59.436262 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27fe81de-7e99-43bd-b130-c699d14180f1 service neutron] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:59.445912 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27fe81de-7e99-43bd-b130-c699d14180f1 service neutron] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:59.500621 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f71d6cb-934b-4566-94fa-4121c4f22148 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:59.501237 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f71d6cb-934b-4566-94fa-4121c4f22148 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:59.501514 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f71d6cb-934b-4566-94fa-4121c4f22148 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:59.501785 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f71d6cb-934b-4566-94fa-4121c4f22148 admin admin] PATH_INFO: `/v3/users/4dee88f12ef84a328565b349a0bc8a49` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:59.502923 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f71d6cb-934b-4566-94fa-4121c4f22148 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4dee88f12ef84a328565b349a0bc8a49)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:59.505125 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f71d6cb-934b-4566-94fa-4121c4f22148 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:59.509725 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:59:59.554826 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-2f71d6cb-934b-4566-94fa-4121c4f22148 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4dee88f12ef84a328565b349a0bc8a49'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:59.555476 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-2f71d6cb-934b-4566-94fa-4121c4f22148 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4dee88f12ef84a328565b349a0bc8a49'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:59.556088 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-2f71d6cb-934b-4566-94fa-4121c4f22148 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4dee88f12ef84a328565b349a0bc8a49'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:59.562852 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-2f71d6cb-934b-4566-94fa-4121c4f22148 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4dee88f12ef84a328565b349a0bc8a49'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:59.574437 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-2f71d6cb-934b-4566-94fa-4121c4f22148 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4dee88f12ef84a328565b349a0bc8a49'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:59.588860 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1356/2721] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 00:59:59 2022] DELETE /identity/v3/users/4dee88f12ef84a328565b349a0bc8a49 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 00:59:59.855281 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef3b6a45-aa97-4305-895a-76f3f397485b admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 00:59:59.856832 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-ef3b6a45-aa97-4305-895a-76f3f397485b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 00:59:59.872817 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef3b6a45-aa97-4305-895a-76f3f397485b admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 00:59:59.890582 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 00:59:59.934141 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf855f1-2cfd-4f13-a06e-6bac9e24093a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 00:59:59.934812 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf855f1-2cfd-4f13-a06e-6bac9e24093a admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 00:59:59.935097 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf855f1-2cfd-4f13-a06e-6bac9e24093a admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 00:59:59.935369 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf855f1-2cfd-4f13-a06e-6bac9e24093a admin admin] PATH_INFO: `/v3/projects/7d515b6e5817481fad25a37e38170dfa` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 00:59:59.936499 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbf855f1-2cfd-4f13-a06e-6bac9e24093a admin admin] RBAC: Authorizing `identity:delete_project(project_id=7d515b6e5817481fad25a37e38170dfa)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 00:59:59.938473 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbf855f1-2cfd-4f13-a06e-6bac9e24093a admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 00:59:59.985607 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-cbf855f1-2cfd-4f13-a06e-6bac9e24093a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7d515b6e5817481fad25a37e38170dfa', 'request_id': 'req-cbf855f1-2cfd-4f13-a06e-6bac9e24093a'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:59.991759 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-cbf855f1-2cfd-4f13-a06e-6bac9e24093a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7d515b6e5817481fad25a37e38170dfa', 'request_id': 'req-cbf855f1-2cfd-4f13-a06e-6bac9e24093a'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 00:59:59.999367 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-cbf855f1-2cfd-4f13-a06e-6bac9e24093a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7d515b6e5817481fad25a37e38170dfa', 'request_id': 'req-cbf855f1-2cfd-4f13-a06e-6bac9e24093a'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:00:00.009857 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-cbf855f1-2cfd-4f13-a06e-6bac9e24093a admin admin] The token cache is being invalidate because project 7d515b6e5817481fad25a37e38170dfa was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87924) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 01:00:00.010139 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-cbf855f1-2cfd-4f13-a06e-6bac9e24093a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:00:00.010752 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-cbf855f1-2cfd-4f13-a06e-6bac9e24093a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:00:00.012243 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1366/2722] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 00:59:59 2022] DELETE /identity/v3/projects/7d515b6e5817481fad25a37e38170dfa => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 01:00:00.018371 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35241ee2-42cb-4146-b9c9-b99298d88e01 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:00:00.018864 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35241ee2-42cb-4146-b9c9-b99298d88e01 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:00:00.018986 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35241ee2-42cb-4146-b9c9-b99298d88e01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:00:00.040166 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-35241ee2-42cb-4146-b9c9-b99298d88e01 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:00:00.041350 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-35241ee2-42cb-4146-b9c9-b99298d88e01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:00:00.100654 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1357/2723] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 01:00:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 01:00:00.108522 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2583fb28-a1ed-483c-8ca6-43fa2ca54f2b None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:00:00.108810 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2583fb28-a1ed-483c-8ca6-43fa2ca54f2b None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:00:00.109119 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2583fb28-a1ed-483c-8ca6-43fa2ca54f2b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:00:00.129358 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-2583fb28-a1ed-483c-8ca6-43fa2ca54f2b None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:00:00.130555 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-2583fb28-a1ed-483c-8ca6-43fa2ca54f2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:00:00.138553 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1367/2724] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 01:00:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 01:00:00.142722 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35241ee2-42cb-4146-b9c9-b99298d88e01 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:00:00.153691 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35241ee2-42cb-4146-b9c9-b99298d88e01 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:00:00.158664 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e3515a-2a42-492c-9d62-d5a5e9cd086f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:00:00.159325 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e3515a-2a42-492c-9d62-d5a5e9cd086f admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:00:00.159603 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e3515a-2a42-492c-9d62-d5a5e9cd086f admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:00:00.159935 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e3515a-2a42-492c-9d62-d5a5e9cd086f admin admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:00:00.161184 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07e3515a-2a42-492c-9d62-d5a5e9cd086f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 01:00:00.161514 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07e3515a-2a42-492c-9d62-d5a5e9cd086f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:00:00.163147 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07e3515a-2a42-492c-9d62-d5a5e9cd086f admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:00:00.174741 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1358/2725] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 01:00:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 01:00:00.179071 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2583fb28-a1ed-483c-8ca6-43fa2ca54f2b None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:00:00.188109 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2583fb28-a1ed-483c-8ca6-43fa2ca54f2b None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:00:00.193473 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa051a2-e3e8-453e-80fc-729aab288181 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:00:00.194069 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa051a2-e3e8-453e-80fc-729aab288181 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:00:00.194351 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa051a2-e3e8-453e-80fc-729aab288181 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:00:00.194619 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa051a2-e3e8-453e-80fc-729aab288181 admin admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:00:00.195473 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fa051a2-e3e8-453e-80fc-729aab288181 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:00:00.197253 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fa051a2-e3e8-453e-80fc-729aab288181 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:00:00.224823 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1368/2726] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 01:00:00 2022] POST /identity/v3/projects => generated 365 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 01:00:00.228827 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e3515a-2a42-492c-9d62-d5a5e9cd086f admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:00:00.238713 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e3515a-2a42-492c-9d62-d5a5e9cd086f admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:00:00.294407 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29421cc1-21e6-4507-8c63-1fe5b66289f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:00:00.295040 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29421cc1-21e6-4507-8c63-1fe5b66289f3 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:00:00.295324 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29421cc1-21e6-4507-8c63-1fe5b66289f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:00:00.295598 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29421cc1-21e6-4507-8c63-1fe5b66289f3 admin admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:00:00.296220 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29421cc1-21e6-4507-8c63-1fe5b66289f3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:00:00.298130 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29421cc1-21e6-4507-8c63-1fe5b66289f3 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:00:00.314066 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:00:00.323083 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1359/2727] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 01:00:00 2022] POST /identity/v3/users => generated 283 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 01:00:00.324242 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa051a2-e3e8-453e-80fc-729aab288181 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:00:00.335830 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa051a2-e3e8-453e-80fc-729aab288181 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:00:00.341280 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e2ee04-9d15-440a-b284-5e16099d88c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:00:00.341886 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e2ee04-9d15-440a-b284-5e16099d88c7 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:00:00.342167 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e2ee04-9d15-440a-b284-5e16099d88c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:00:00.342439 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e2ee04-9d15-440a-b284-5e16099d88c7 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:00:00.343121 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57e2ee04-9d15-440a-b284-5e16099d88c7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:00:00.344719 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57e2ee04-9d15-440a-b284-5e16099d88c7 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:00:00.355154 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1369/2728] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 01:00:00 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 11 01:00:00.359296 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29421cc1-21e6-4507-8c63-1fe5b66289f3 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:00:00.369333 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29421cc1-21e6-4507-8c63-1fe5b66289f3 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:00:00.373992 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0ac698-f479-4230-a543-f8b09432f488 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:00:00.374583 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0ac698-f479-4230-a543-f8b09432f488 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:00:00.374897 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0ac698-f479-4230-a543-f8b09432f488 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:00:00.375174 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0ac698-f479-4230-a543-f8b09432f488 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:00:00.375821 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d0ac698-f479-4230-a543-f8b09432f488 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:00:00.377414 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d0ac698-f479-4230-a543-f8b09432f488 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:00:00.388732 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1360/2729] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 01:00:00 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 01:00:00.392942 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e2ee04-9d15-440a-b284-5e16099d88c7 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:00:00.405215 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e2ee04-9d15-440a-b284-5e16099d88c7 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:00:00.410312 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b3791d-e29f-4f72-8b1c-ab4cc2ac64ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:00:00.411194 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b3791d-e29f-4f72-8b1c-ab4cc2ac64ee admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:00:00.411477 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b3791d-e29f-4f72-8b1c-ab4cc2ac64ee admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:00:00.411754 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b3791d-e29f-4f72-8b1c-ab4cc2ac64ee admin admin] PATH_INFO: `/v3/projects/7a1656c77f334bde9f4bea96616ead06/users/48376c7a59e84455a171954d966e2813/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:00:00.427281 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38b3791d-e29f-4f72-8b1c-ab4cc2ac64ee admin admin] RBAC: Authorizing `identity:create_grant(project_id=7a1656c77f334bde9f4bea96616ead06, user_id=48376c7a59e84455a171954d966e2813, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:00:00.429922 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38b3791d-e29f-4f72-8b1c-ab4cc2ac64ee admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:00:00.443063 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1370/2730] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 01:00:00 2022] PUT /identity/v3/projects/7a1656c77f334bde9f4bea96616ead06/users/48376c7a59e84455a171954d966e2813/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 01:00:00.454271 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05feb08a-201b-40a9-a895-3a5aec37ac36 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:00:00.454584 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05feb08a-201b-40a9-a895-3a5aec37ac36 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:00:00.454891 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05feb08a-201b-40a9-a895-3a5aec37ac36 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:00:00.482610 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-05feb08a-201b-40a9-a895-3a5aec37ac36 None None] MFA Rules not processed for user `48376c7a59e84455a171954d966e2813`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:00:00.483824 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-05feb08a-201b-40a9-a895-3a5aec37ac36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:00:00.549004 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1361/2731] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 01:00:00 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 01:00:00.560721 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b3791d-e29f-4f72-8b1c-ab4cc2ac64ee admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:00:00.562457 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-38b3791d-e29f-4f72-8b1c-ab4cc2ac64ee admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:00:00.614305 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:00:00.620504 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b3791d-e29f-4f72-8b1c-ab4cc2ac64ee admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:00:00.625165 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a41aa7-ae0a-492c-aaac-21dfd7d277a5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:00:00.625771 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a41aa7-ae0a-492c-aaac-21dfd7d277a5 service cinder] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:00:00.626050 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a41aa7-ae0a-492c-aaac-21dfd7d277a5 service cinder] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:00:00.626321 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a41aa7-ae0a-492c-aaac-21dfd7d277a5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:00:00.635324 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a41aa7-ae0a-492c-aaac-21dfd7d277a5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:00:00.636770 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a41aa7-ae0a-492c-aaac-21dfd7d277a5 service cinder] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:00:00.645583 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1371/2732] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 01:00:00 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 01:00:02.018549 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f078898-188a-4949-8ffc-b3e380deb083 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:00:02.019072 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f078898-188a-4949-8ffc-b3e380deb083 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:00:02.019248 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f078898-188a-4949-8ffc-b3e380deb083 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:00:02.039177 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-0f078898-188a-4949-8ffc-b3e380deb083 None None] MFA Rules not processed for user `78a2199d3f1d4bc5b1f988ae06c0d609`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:00:02.040363 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-0f078898-188a-4949-8ffc-b3e380deb083 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:00:02.097780 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1362/2733] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 01:00:02 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 01:00:10.641453 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=5 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:00:12.090431 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=62 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:00:20.874101 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a41aa7-ae0a-492c-aaac-21dfd7d277a5 service cinder] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:00:20.875573 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-d6a41aa7-ae0a-492c-aaac-21dfd7d277a5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:00:20.934603 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a41aa7-ae0a-492c-aaac-21dfd7d277a5 service cinder] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:00:20.938283 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e053ab-a4ef-4f98-a726-78695afaf5df service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:00:20.938925 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e053ab-a4ef-4f98-a726-78695afaf5df service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:00:20.939207 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e053ab-a4ef-4f98-a726-78695afaf5df service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:00:20.939479 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e053ab-a4ef-4f98-a726-78695afaf5df service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:00:20.941222 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-d0e053ab-a4ef-4f98-a726-78695afaf5df service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:00:21.014776 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e053ab-a4ef-4f98-a726-78695afaf5df service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:00:21.016603 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e053ab-a4ef-4f98-a726-78695afaf5df service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:00:21.026427 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1372/2734] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 01:00:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 153 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 01:00:24.108722 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f078898-188a-4949-8ffc-b3e380deb083 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:00:24.110736 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-0f078898-188a-4949-8ffc-b3e380deb083 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:00:24.149167 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f078898-188a-4949-8ffc-b3e380deb083 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:00:24.154474 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e77ccc-44fd-4080-9228-e883a58582d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:00:24.155269 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e77ccc-44fd-4080-9228-e883a58582d7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:00:24.155546 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e77ccc-44fd-4080-9228-e883a58582d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:00:24.155816 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e77ccc-44fd-4080-9228-e883a58582d7 admin admin] PATH_INFO: `/v3/users/73a3d3c48913474ab8fa8fedd1bc7d03` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:00:24.157280 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81e77ccc-44fd-4080-9228-e883a58582d7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=73a3d3c48913474ab8fa8fedd1bc7d03)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:00:24.159236 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81e77ccc-44fd-4080-9228-e883a58582d7 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:00:24.217861 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-81e77ccc-44fd-4080-9228-e883a58582d7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '73a3d3c48913474ab8fa8fedd1bc7d03'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:00:24.218527 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-81e77ccc-44fd-4080-9228-e883a58582d7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '73a3d3c48913474ab8fa8fedd1bc7d03'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:00:24.219165 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-81e77ccc-44fd-4080-9228-e883a58582d7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '73a3d3c48913474ab8fa8fedd1bc7d03'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:00:24.234837 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-81e77ccc-44fd-4080-9228-e883a58582d7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '73a3d3c48913474ab8fa8fedd1bc7d03'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:00:24.256438 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-81e77ccc-44fd-4080-9228-e883a58582d7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '73a3d3c48913474ab8fa8fedd1bc7d03'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:00:24.274966 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1363/2735] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 01:00:24 2022] DELETE /identity/v3/users/73a3d3c48913474ab8fa8fedd1bc7d03 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 01:00:24.477705 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e053ab-a4ef-4f98-a726-78695afaf5df service neutron] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:00:24.489002 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e053ab-a4ef-4f98-a726-78695afaf5df service neutron] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:00:24.548510 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e895f550-d357-45e1-a01e-090fe5dba158 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:00:24.549149 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e895f550-d357-45e1-a01e-090fe5dba158 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:00:24.549430 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e895f550-d357-45e1-a01e-090fe5dba158 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:00:24.549709 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e895f550-d357-45e1-a01e-090fe5dba158 admin admin] PATH_INFO: `/v3/projects/fd9221b2e81a46f69864e02ec4d42672` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:00:24.551520 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e895f550-d357-45e1-a01e-090fe5dba158 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fd9221b2e81a46f69864e02ec4d42672)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:00:24.553706 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e895f550-d357-45e1-a01e-090fe5dba158 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:00:24.605057 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-e895f550-d357-45e1-a01e-090fe5dba158 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fd9221b2e81a46f69864e02ec4d42672', 'request_id': 'req-e895f550-d357-45e1-a01e-090fe5dba158'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:00:24.606874 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:00:24.612036 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-e895f550-d357-45e1-a01e-090fe5dba158 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fd9221b2e81a46f69864e02ec4d42672', 'request_id': 'req-e895f550-d357-45e1-a01e-090fe5dba158'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:00:24.620238 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-e895f550-d357-45e1-a01e-090fe5dba158 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fd9221b2e81a46f69864e02ec4d42672', 'request_id': 'req-e895f550-d357-45e1-a01e-090fe5dba158'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:00:24.631040 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-e895f550-d357-45e1-a01e-090fe5dba158 admin admin] The token cache is being invalidate because project fd9221b2e81a46f69864e02ec4d42672 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87924) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 01:00:24.631457 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-e895f550-d357-45e1-a01e-090fe5dba158 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:00:24.632111 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-e895f550-d357-45e1-a01e-090fe5dba158 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:00:24.633595 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1373/2736] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 01:00:24 2022] DELETE /identity/v3/projects/fd9221b2e81a46f69864e02ec4d42672 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 01:00:24.638610 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282a2acb-3066-4392-a409-ee43e4db2cf1 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:00:24.638937 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282a2acb-3066-4392-a409-ee43e4db2cf1 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:00:24.639213 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282a2acb-3066-4392-a409-ee43e4db2cf1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:00:24.660020 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-282a2acb-3066-4392-a409-ee43e4db2cf1 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:00:24.661899 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-282a2acb-3066-4392-a409-ee43e4db2cf1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:00:24.720972 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1364/2737] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 01:00:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 01:00:24.729721 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6c327d-02af-4334-b182-406a90fa7dea None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:00:24.730003 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6c327d-02af-4334-b182-406a90fa7dea None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:00:24.730274 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6c327d-02af-4334-b182-406a90fa7dea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:00:24.750742 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-3d6c327d-02af-4334-b182-406a90fa7dea None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:00:24.751967 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-3d6c327d-02af-4334-b182-406a90fa7dea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:00:24.761083 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1374/2738] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 01:00:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 01:00:24.765477 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-282a2acb-3066-4392-a409-ee43e4db2cf1 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:00:24.775666 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-282a2acb-3066-4392-a409-ee43e4db2cf1 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:00:24.780278 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5088e1f-e015-4078-89c8-650b51187d0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:00:24.781144 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5088e1f-e015-4078-89c8-650b51187d0f admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:00:24.781426 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5088e1f-e015-4078-89c8-650b51187d0f admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:00:24.781699 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5088e1f-e015-4078-89c8-650b51187d0f admin admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:00:24.782349 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5088e1f-e015-4078-89c8-650b51187d0f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 01:00:24.782654 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5088e1f-e015-4078-89c8-650b51187d0f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:00:24.784302 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5088e1f-e015-4078-89c8-650b51187d0f admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:00:24.796453 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1365/2739] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 01:00:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 01:00:24.802100 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d6c327d-02af-4334-b182-406a90fa7dea None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:00:24.811683 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d6c327d-02af-4334-b182-406a90fa7dea None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:00:24.816278 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af47b96-ec27-44de-8774-1b2325566c43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:00:24.817120 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af47b96-ec27-44de-8774-1b2325566c43 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:00:24.817425 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af47b96-ec27-44de-8774-1b2325566c43 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:00:24.817695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af47b96-ec27-44de-8774-1b2325566c43 admin admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:00:24.818319 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7af47b96-ec27-44de-8774-1b2325566c43 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:00:24.820232 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7af47b96-ec27-44de-8774-1b2325566c43 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:00:24.848592 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1375/2740] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 01:00:24 2022] POST /identity/v3/projects => generated 377 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 01:00:24.853109 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5088e1f-e015-4078-89c8-650b51187d0f admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:00:24.864011 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5088e1f-e015-4078-89c8-650b51187d0f admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:00:24.923296 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da81999b-2671-4af1-a54b-ee6eb3d9d1f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:00:24.924030 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da81999b-2671-4af1-a54b-ee6eb3d9d1f4 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:00:24.924309 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da81999b-2671-4af1-a54b-ee6eb3d9d1f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:00:24.924582 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da81999b-2671-4af1-a54b-ee6eb3d9d1f4 admin admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:00:24.925216 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da81999b-2671-4af1-a54b-ee6eb3d9d1f4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:00:24.927193 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da81999b-2671-4af1-a54b-ee6eb3d9d1f4 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:00:24.935975 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:00:24.951968 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1366/2741] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 01:00:24 2022] POST /identity/v3/users => generated 289 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 01:00:24.956294 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af47b96-ec27-44de-8774-1b2325566c43 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:00:24.968365 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af47b96-ec27-44de-8774-1b2325566c43 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:00:24.974086 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd945b57-13b2-44c5-a57d-b1f7ef169fa3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:00:24.974747 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd945b57-13b2-44c5-a57d-b1f7ef169fa3 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:00:24.975053 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd945b57-13b2-44c5-a57d-b1f7ef169fa3 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:00:24.975322 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd945b57-13b2-44c5-a57d-b1f7ef169fa3 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:00:24.975962 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd945b57-13b2-44c5-a57d-b1f7ef169fa3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:00:24.977577 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd945b57-13b2-44c5-a57d-b1f7ef169fa3 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:00:24.987852 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1376/2742] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 01:00:24 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 01:00:24.992347 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da81999b-2671-4af1-a54b-ee6eb3d9d1f4 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:00:25.002380 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da81999b-2671-4af1-a54b-ee6eb3d9d1f4 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:00:25.007157 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf6ee043-9f94-45b8-85a2-e4317b53842a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:00:25.007740 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf6ee043-9f94-45b8-85a2-e4317b53842a admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:00:25.008018 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf6ee043-9f94-45b8-85a2-e4317b53842a admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:00:25.008284 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf6ee043-9f94-45b8-85a2-e4317b53842a admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:00:25.009006 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf6ee043-9f94-45b8-85a2-e4317b53842a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:00:25.010816 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf6ee043-9f94-45b8-85a2-e4317b53842a admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:00:25.023828 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1367/2743] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 01:00:24 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 01:00:25.028175 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd945b57-13b2-44c5-a57d-b1f7ef169fa3 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:00:25.039154 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd945b57-13b2-44c5-a57d-b1f7ef169fa3 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:00:25.043868 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f7885f6-f500-462f-8aba-aef67645da3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:00:25.044512 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f7885f6-f500-462f-8aba-aef67645da3d admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:00:25.044791 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f7885f6-f500-462f-8aba-aef67645da3d admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:00:25.045063 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f7885f6-f500-462f-8aba-aef67645da3d admin admin] PATH_INFO: `/v3/projects/15810c36aaa44c5fbc87197f873a6260/users/58b3528fd06b4d438caa92fd88383ed5/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:00:25.060970 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f7885f6-f500-462f-8aba-aef67645da3d admin admin] RBAC: Authorizing `identity:create_grant(project_id=15810c36aaa44c5fbc87197f873a6260, user_id=58b3528fd06b4d438caa92fd88383ed5, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:00:25.063673 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f7885f6-f500-462f-8aba-aef67645da3d admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:00:25.076339 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1377/2744] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 01:00:25 2022] PUT /identity/v3/projects/15810c36aaa44c5fbc87197f873a6260/users/58b3528fd06b4d438caa92fd88383ed5/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 01:00:25.081730 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e2f831-48c6-445c-a4d9-e1b9e66b3f21 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:00:25.082617 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e2f831-48c6-445c-a4d9-e1b9e66b3f21 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:00:25.082966 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e2f831-48c6-445c-a4d9-e1b9e66b3f21 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:00:25.102978 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-74e2f831-48c6-445c-a4d9-e1b9e66b3f21 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:00:25.104222 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-74e2f831-48c6-445c-a4d9-e1b9e66b3f21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:00:25.164992 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1368/2745] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 01:00:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 01:00:25.174986 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f7885f6-f500-462f-8aba-aef67645da3d admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:00:25.176972 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-6f7885f6-f500-462f-8aba-aef67645da3d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:00:25.234782 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f7885f6-f500-462f-8aba-aef67645da3d admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:00:25.238485 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462ed3c8-b61a-4ca0-84f7-92e6668fa7c7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:00:25.239138 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462ed3c8-b61a-4ca0-84f7-92e6668fa7c7 service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:00:25.239412 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462ed3c8-b61a-4ca0-84f7-92e6668fa7c7 service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:00:25.239681 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462ed3c8-b61a-4ca0-84f7-92e6668fa7c7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:00:25.248827 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-462ed3c8-b61a-4ca0-84f7-92e6668fa7c7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:00:25.250555 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-462ed3c8-b61a-4ca0-84f7-92e6668fa7c7 service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:00:25.260388 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1378/2746] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 01:00:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 01:00:29.451000 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff5839f-8fe1-4114-97c3-ff57a780d227 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:00:29.451563 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff5839f-8fe1-4114-97c3-ff57a780d227 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:00:29.451671 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff5839f-8fe1-4114-97c3-ff57a780d227 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:00:29.493921 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-cff5839f-8fe1-4114-97c3-ff57a780d227 None None] MFA Rules not processed for user `58b3528fd06b4d438caa92fd88383ed5`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:00:29.495234 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-cff5839f-8fe1-4114-97c3-ff57a780d227 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:00:29.569362 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1369/2747] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 01:00:29 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 01:00:29.584981 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462ed3c8-b61a-4ca0-84f7-92e6668fa7c7 service neutron] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:00:29.586958 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-462ed3c8-b61a-4ca0-84f7-92e6668fa7c7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:00:29.668479 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462ed3c8-b61a-4ca0-84f7-92e6668fa7c7 service neutron] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:00:29.674008 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b87fe698-a6c3-4d3d-a142-ad05da59bd27 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '835def0e6c6d4365a2767051509eb0d6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'ResellerAdmin', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:00:29.674634 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b87fe698-a6c3-4d3d-a142-ad05da59bd27 service nova] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:00:29.674946 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b87fe698-a6c3-4d3d-a142-ad05da59bd27 service nova] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:00:29.675208 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b87fe698-a6c3-4d3d-a142-ad05da59bd27 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:00:29.683748 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b87fe698-a6c3-4d3d-a142-ad05da59bd27 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:00:29.685442 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b87fe698-a6c3-4d3d-a142-ad05da59bd27 service nova] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:00:29.695286 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1379/2748] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 01:00:29 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 01:00:31.057211 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3fcfd67-7eed-40b0-b351-2fc1a3f39a28 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:00:31.057760 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3fcfd67-7eed-40b0-b351-2fc1a3f39a28 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:00:31.057860 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3fcfd67-7eed-40b0-b351-2fc1a3f39a28 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:00:31.079149 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-c3fcfd67-7eed-40b0-b351-2fc1a3f39a28 None None] MFA Rules not processed for user `58b3528fd06b4d438caa92fd88383ed5`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:00:31.080284 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-c3fcfd67-7eed-40b0-b351-2fc1a3f39a28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:00:31.088182 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1370/2749] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 01:00:31 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 01:00:39.690624 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=92,keystone:INSERT=3,keystone:DELETE=3 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:00:41.072108 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:INSERT=3,keystone:SELECT=69 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:00:41.388603 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b87fe698-a6c3-4d3d-a142-ad05da59bd27 service nova] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:00:41.390016 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-b87fe698-a6c3-4d3d-a142-ad05da59bd27 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:00:41.468056 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b87fe698-a6c3-4d3d-a142-ad05da59bd27 service nova] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:00:41.472725 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4878436-afa5-42bb-8b03-33d426e9e348 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:00:41.473519 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4878436-afa5-42bb-8b03-33d426e9e348 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:00:41.473798 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4878436-afa5-42bb-8b03-33d426e9e348 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:00:41.474122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4878436-afa5-42bb-8b03-33d426e9e348 admin admin] PATH_INFO: `/v3/users/48376c7a59e84455a171954d966e2813` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:00:41.475267 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4878436-afa5-42bb-8b03-33d426e9e348 admin admin] RBAC: Authorizing `identity:delete_user(user_id=48376c7a59e84455a171954d966e2813)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:00:41.477253 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4878436-afa5-42bb-8b03-33d426e9e348 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:00:41.532481 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-c4878436-afa5-42bb-8b03-33d426e9e348 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '48376c7a59e84455a171954d966e2813'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:00:41.533227 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-c4878436-afa5-42bb-8b03-33d426e9e348 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '48376c7a59e84455a171954d966e2813'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:00:41.533849 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-c4878436-afa5-42bb-8b03-33d426e9e348 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '48376c7a59e84455a171954d966e2813'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:00:41.541475 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-c4878436-afa5-42bb-8b03-33d426e9e348 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '48376c7a59e84455a171954d966e2813'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:00:41.554146 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-c4878436-afa5-42bb-8b03-33d426e9e348 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '48376c7a59e84455a171954d966e2813'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:00:41.571495 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1380/2750] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 01:00:41 2022] DELETE /identity/v3/users/48376c7a59e84455a171954d966e2813 => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 01:00:41.576165 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ac611a-e107-488e-b152-7ec043fab630 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:00:41.576483 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ac611a-e107-488e-b152-7ec043fab630 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:00:41.576761 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ac611a-e107-488e-b152-7ec043fab630 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:00:41.609586 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-d8ac611a-e107-488e-b152-7ec043fab630 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:00:41.611271 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-d8ac611a-e107-488e-b152-7ec043fab630 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:00:41.683097 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1371/2751] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 01:00:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 01:00:41.689609 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4878436-afa5-42bb-8b03-33d426e9e348 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:00:41.691094 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-c4878436-afa5-42bb-8b03-33d426e9e348 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:00:41.772180 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4878436-afa5-42bb-8b03-33d426e9e348 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:00:41.775906 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59ff9f56-6e88-4d2c-bd7a-0e5c907d1dac service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:00:41.776503 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ff9f56-6e88-4d2c-bd7a-0e5c907d1dac service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:00:41.776782 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ff9f56-6e88-4d2c-bd7a-0e5c907d1dac service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:00:41.777055 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ff9f56-6e88-4d2c-bd7a-0e5c907d1dac service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:00:41.787224 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59ff9f56-6e88-4d2c-bd7a-0e5c907d1dac service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:00:41.788657 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59ff9f56-6e88-4d2c-bd7a-0e5c907d1dac service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:00:41.792681 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:00:41.799991 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1381/2752] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 01:00:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 01:00:42.085109 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8ac611a-e107-488e-b152-7ec043fab630 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:00:42.087833 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-d8ac611a-e107-488e-b152-7ec043fab630 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:00:42.106044 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8ac611a-e107-488e-b152-7ec043fab630 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:00:42.113004 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be280c3-c546-4a53-8372-9827dabd8126 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:00:42.115216 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be280c3-c546-4a53-8372-9827dabd8126 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:00:42.115971 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be280c3-c546-4a53-8372-9827dabd8126 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:00:42.116627 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be280c3-c546-4a53-8372-9827dabd8126 admin admin] PATH_INFO: `/v3/projects/7a1656c77f334bde9f4bea96616ead06` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:00:42.117889 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1be280c3-c546-4a53-8372-9827dabd8126 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7a1656c77f334bde9f4bea96616ead06)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:00:42.120133 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1be280c3-c546-4a53-8372-9827dabd8126 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:00:42.169787 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-1be280c3-c546-4a53-8372-9827dabd8126 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7a1656c77f334bde9f4bea96616ead06', 'request_id': 'req-1be280c3-c546-4a53-8372-9827dabd8126'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:00:42.178971 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-1be280c3-c546-4a53-8372-9827dabd8126 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7a1656c77f334bde9f4bea96616ead06', 'request_id': 'req-1be280c3-c546-4a53-8372-9827dabd8126'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:00:42.184809 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-1be280c3-c546-4a53-8372-9827dabd8126 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7a1656c77f334bde9f4bea96616ead06', 'request_id': 'req-1be280c3-c546-4a53-8372-9827dabd8126'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:00:42.199164 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-1be280c3-c546-4a53-8372-9827dabd8126 admin admin] The token cache is being invalidate because project 7a1656c77f334bde9f4bea96616ead06 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87923) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 01:00:42.199570 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-1be280c3-c546-4a53-8372-9827dabd8126 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:00:42.200209 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-1be280c3-c546-4a53-8372-9827dabd8126 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:00:42.202009 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1372/2753] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 01:00:42 2022] DELETE /identity/v3/projects/7a1656c77f334bde9f4bea96616ead06 => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 01:00:42.206632 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa82c2aa-8842-464f-87ce-e9531cf15d42 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:00:42.207026 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa82c2aa-8842-464f-87ce-e9531cf15d42 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:00:42.207387 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa82c2aa-8842-464f-87ce-e9531cf15d42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:00:42.265611 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-fa82c2aa-8842-464f-87ce-e9531cf15d42 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:00:42.266986 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-fa82c2aa-8842-464f-87ce-e9531cf15d42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:00:42.334771 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1382/2754] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 01:00:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 01:00:42.342498 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616fdba1-a374-489e-9037-3e50982abefe None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:00:42.342942 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616fdba1-a374-489e-9037-3e50982abefe None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:00:42.343132 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616fdba1-a374-489e-9037-3e50982abefe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:00:42.363428 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-616fdba1-a374-489e-9037-3e50982abefe None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:00:42.365203 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-616fdba1-a374-489e-9037-3e50982abefe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:00:42.373277 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1373/2755] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 01:00:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 01:00:42.377491 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa82c2aa-8842-464f-87ce-e9531cf15d42 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:00:42.388143 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa82c2aa-8842-464f-87ce-e9531cf15d42 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:00:42.392941 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f997d30d-a954-4484-b59d-44f04558365b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:00:42.393564 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f997d30d-a954-4484-b59d-44f04558365b admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:00:42.393808 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f997d30d-a954-4484-b59d-44f04558365b admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:00:42.394078 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f997d30d-a954-4484-b59d-44f04558365b admin admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:00:42.394716 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f997d30d-a954-4484-b59d-44f04558365b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 01:00:42.395121 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f997d30d-a954-4484-b59d-44f04558365b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:00:42.396917 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f997d30d-a954-4484-b59d-44f04558365b admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:00:42.410933 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1383/2756] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 01:00:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 01:00:42.416003 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-616fdba1-a374-489e-9037-3e50982abefe None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:00:42.426334 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-616fdba1-a374-489e-9037-3e50982abefe None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:00:42.432203 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed9a9f3-0f59-4aaa-a81b-5aac4df96ba6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:00:42.432828 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed9a9f3-0f59-4aaa-a81b-5aac4df96ba6 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:00:42.433101 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed9a9f3-0f59-4aaa-a81b-5aac4df96ba6 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:00:42.433366 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed9a9f3-0f59-4aaa-a81b-5aac4df96ba6 admin admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:00:42.433992 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed9a9f3-0f59-4aaa-a81b-5aac4df96ba6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:00:42.435949 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed9a9f3-0f59-4aaa-a81b-5aac4df96ba6 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:00:42.464469 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1374/2757] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 01:00:42 2022] POST /identity/v3/projects => generated 373 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 01:00:42.468689 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f997d30d-a954-4484-b59d-44f04558365b admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:00:42.478526 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f997d30d-a954-4484-b59d-44f04558365b admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:00:42.536678 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cba65e6b-e8c0-47ba-a7e4-5c5299413a20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:00:42.537578 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba65e6b-e8c0-47ba-a7e4-5c5299413a20 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:00:42.537856 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba65e6b-e8c0-47ba-a7e4-5c5299413a20 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:00:42.538124 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba65e6b-e8c0-47ba-a7e4-5c5299413a20 admin admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:00:42.538736 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cba65e6b-e8c0-47ba-a7e4-5c5299413a20 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:00:42.540697 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cba65e6b-e8c0-47ba-a7e4-5c5299413a20 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:00:42.564009 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1384/2758] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 01:00:42 2022] POST /identity/v3/users => generated 287 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 01:00:42.567765 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed9a9f3-0f59-4aaa-a81b-5aac4df96ba6 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:00:42.582505 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed9a9f3-0f59-4aaa-a81b-5aac4df96ba6 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:00:42.590906 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-489c8aa9-1130-4063-aa61-64366f160adc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:00:42.591530 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489c8aa9-1130-4063-aa61-64366f160adc admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:00:42.591861 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489c8aa9-1130-4063-aa61-64366f160adc admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:00:42.592134 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489c8aa9-1130-4063-aa61-64366f160adc admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:00:42.592788 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-489c8aa9-1130-4063-aa61-64366f160adc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:00:42.594457 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-489c8aa9-1130-4063-aa61-64366f160adc admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:00:42.606360 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1375/2759] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 01:00:42 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 01:00:42.611035 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cba65e6b-e8c0-47ba-a7e4-5c5299413a20 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:00:42.625015 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cba65e6b-e8c0-47ba-a7e4-5c5299413a20 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:00:42.629698 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aea23cd-68d9-4aba-a2d1-0fe6644fa933 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:00:42.630459 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aea23cd-68d9-4aba-a2d1-0fe6644fa933 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:00:42.630742 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aea23cd-68d9-4aba-a2d1-0fe6644fa933 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:00:42.631087 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aea23cd-68d9-4aba-a2d1-0fe6644fa933 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:00:42.631758 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aea23cd-68d9-4aba-a2d1-0fe6644fa933 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:00:42.633361 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aea23cd-68d9-4aba-a2d1-0fe6644fa933 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:00:42.643986 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1385/2760] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 01:00:42 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 01:00:42.648208 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-489c8aa9-1130-4063-aa61-64366f160adc admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:00:42.658481 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-489c8aa9-1130-4063-aa61-64366f160adc admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:00:42.663733 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa2e3b3-c42c-4408-bfce-0242d0a26c34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:00:42.664377 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa2e3b3-c42c-4408-bfce-0242d0a26c34 admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:00:42.664655 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa2e3b3-c42c-4408-bfce-0242d0a26c34 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:00:42.664929 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa2e3b3-c42c-4408-bfce-0242d0a26c34 admin admin] PATH_INFO: `/v3/projects/d99b9dcee07048b49f75a4e0172b42be/users/e0a15a4c53da4903a3f1444db955ab79/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:00:42.679918 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aa2e3b3-c42c-4408-bfce-0242d0a26c34 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d99b9dcee07048b49f75a4e0172b42be, user_id=e0a15a4c53da4903a3f1444db955ab79, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:00:42.682549 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aa2e3b3-c42c-4408-bfce-0242d0a26c34 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:00:42.694993 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1376/2761] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 01:00:42 2022] PUT /identity/v3/projects/d99b9dcee07048b49f75a4e0172b42be/users/e0a15a4c53da4903a3f1444db955ab79/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 01:00:42.760679 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66aaf11a-ede3-45ca-997b-70715e1d4790 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:00:42.760962 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66aaf11a-ede3-45ca-997b-70715e1d4790 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:00:42.761251 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66aaf11a-ede3-45ca-997b-70715e1d4790 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:00:42.789853 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-66aaf11a-ede3-45ca-997b-70715e1d4790 None None] MFA Rules not processed for user `e0a15a4c53da4903a3f1444db955ab79`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:00:42.791075 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-66aaf11a-ede3-45ca-997b-70715e1d4790 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:00:42.819769 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:00:42.855620 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1386/2762] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 01:00:42 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 01:00:42.866641 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa2e3b3-c42c-4408-bfce-0242d0a26c34 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:00:42.868602 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-8aa2e3b3-c42c-4408-bfce-0242d0a26c34 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:00:42.882951 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:00:42.926379 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa2e3b3-c42c-4408-bfce-0242d0a26c34 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:00:42.930275 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2aaf6ad-37e0-43db-b3d7-4d14bcb93060 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:00:42.930905 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2aaf6ad-37e0-43db-b3d7-4d14bcb93060 service cinder] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:00:42.931186 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2aaf6ad-37e0-43db-b3d7-4d14bcb93060 service cinder] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:00:42.931456 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2aaf6ad-37e0-43db-b3d7-4d14bcb93060 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:00:42.939955 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2aaf6ad-37e0-43db-b3d7-4d14bcb93060 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:00:42.941378 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2aaf6ad-37e0-43db-b3d7-4d14bcb93060 service cinder] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:00:42.950492 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1377/2763] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 01:00:42 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 01:00:44.344687 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96680eb8-f300-4436-bcfa-33d8e611c967 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:00:44.344687 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96680eb8-f300-4436-bcfa-33d8e611c967 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:00:44.344687 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96680eb8-f300-4436-bcfa-33d8e611c967 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:00:44.373354 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-96680eb8-f300-4436-bcfa-33d8e611c967 None None] MFA Rules not processed for user `e0a15a4c53da4903a3f1444db955ab79`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:00:44.374567 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-96680eb8-f300-4436-bcfa-33d8e611c967 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:00:44.381855 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1387/2764] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 01:00:44 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 01:00:47.593402 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2aaf6ad-37e0-43db-b3d7-4d14bcb93060 service cinder] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:00:47.594820 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-b2aaf6ad-37e0-43db-b3d7-4d14bcb93060 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:00:47.606610 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2aaf6ad-37e0-43db-b3d7-4d14bcb93060 service cinder] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:00:47.610205 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ec92dc-eb49-401d-93a9-cb5a0dbecf97 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:00:47.610808 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ec92dc-eb49-401d-93a9-cb5a0dbecf97 service cinder] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:00:47.611092 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ec92dc-eb49-401d-93a9-cb5a0dbecf97 service cinder] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:00:47.611359 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ec92dc-eb49-401d-93a9-cb5a0dbecf97 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:00:47.613001 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-20ec92dc-eb49-401d-93a9-cb5a0dbecf97 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:00:47.671201 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20ec92dc-eb49-401d-93a9-cb5a0dbecf97 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:00:47.672627 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20ec92dc-eb49-401d-93a9-cb5a0dbecf97 service cinder] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:00:47.682288 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1378/2765] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 01:00:47 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 01:00:52.996390 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96680eb8-f300-4436-bcfa-33d8e611c967 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:00:52.998267 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-96680eb8-f300-4436-bcfa-33d8e611c967 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:00:53.103496 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96680eb8-f300-4436-bcfa-33d8e611c967 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:00:53.109170 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79420339-5bdf-4314-9f6f-69310b4f9774 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cdbfc2ccd73e4a5eb8c999be5b094619', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:00:53.109787 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79420339-5bdf-4314-9f6f-69310b4f9774 service placement] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:00:53.110064 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79420339-5bdf-4314-9f6f-69310b4f9774 service placement] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:00:53.110341 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79420339-5bdf-4314-9f6f-69310b4f9774 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:00:53.112141 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-79420339-5bdf-4314-9f6f-69310b4f9774 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:00:53.124174 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79420339-5bdf-4314-9f6f-69310b4f9774 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:00:53.125762 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79420339-5bdf-4314-9f6f-69310b4f9774 service placement] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:00:53.136738 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1388/2766] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 01:00:52 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 01:00:55.774782 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c857df58-3799-40fd-b092-e67c390980ef None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:00:55.775340 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c857df58-3799-40fd-b092-e67c390980ef None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:00:55.775340 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c857df58-3799-40fd-b092-e67c390980ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:00:55.795790 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-c857df58-3799-40fd-b092-e67c390980ef None None] MFA Rules not processed for user `e1012f856881485cb40d916131a5d763`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:00:55.796998 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-c857df58-3799-40fd-b092-e67c390980ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:00:55.804734 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1379/2767] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 01:00:55 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 01:01:03.132998 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=58 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:01:05.790112 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=51 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:01:06.191487 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2602a23c-17cf-4311-8cde-0d477fd2d133 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:01:06.191968 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2602a23c-17cf-4311-8cde-0d477fd2d133 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:01:06.192201 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2602a23c-17cf-4311-8cde-0d477fd2d133 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:01:06.224524 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-2602a23c-17cf-4311-8cde-0d477fd2d133 None None] MFA Rules not processed for user `9fc4274836844ae4ad5d16d74c36f671`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:01:06.225765 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-2602a23c-17cf-4311-8cde-0d477fd2d133 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:01:06.291091 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1389/2768] 10.222.0.27 () {58 vars in 1009 bytes} [Sun Sep 11 01:01:06 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 01:01:06.336671 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c857df58-3799-40fd-b092-e67c390980ef None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:01:06.338092 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-c857df58-3799-40fd-b092-e67c390980ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:01:06.405092 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c857df58-3799-40fd-b092-e67c390980ef None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:01:06.410221 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e23d1bb6-d825-417a-b9da-e0f4a3240e02 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bcb73dec73464a3abc4b774abb4f416f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:01:06.410835 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23d1bb6-d825-417a-b9da-e0f4a3240e02 service swift] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:01:06.411114 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23d1bb6-d825-417a-b9da-e0f4a3240e02 service swift] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:01:06.412001 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23d1bb6-d825-417a-b9da-e0f4a3240e02 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:01:06.419815 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e23d1bb6-d825-417a-b9da-e0f4a3240e02 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:01:06.421492 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e23d1bb6-d825-417a-b9da-e0f4a3240e02 service swift] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:01:06.429377 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1380/2769] 10.222.0.27 () {60 vars in 1384 bytes} [Sun Sep 11 01:01:06 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 11 01:01:06.946910 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff0448a-f06d-4586-ba4c-b8b4a670140b None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:01:06.947200 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff0448a-f06d-4586-ba4c-b8b4a670140b None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:01:06.947475 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff0448a-f06d-4586-ba4c-b8b4a670140b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:01:06.968130 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-cff0448a-f06d-4586-ba4c-b8b4a670140b None None] MFA Rules not processed for user `9fc4274836844ae4ad5d16d74c36f671`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:01:06.969329 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-cff0448a-f06d-4586-ba4c-b8b4a670140b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:01:06.977201 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1390/2770] 10.222.0.27 () {58 vars in 967 bytes} [Sun Sep 11 01:01:06 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 01:01:06.989448 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e23d1bb6-d825-417a-b9da-e0f4a3240e02 service swift] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:01:07.000790 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e23d1bb6-d825-417a-b9da-e0f4a3240e02 service swift] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:01:07.005918 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a91555-c826-4d4c-9fe3-5fb3019e5bd0 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bcb73dec73464a3abc4b774abb4f416f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:01:07.006547 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a91555-c826-4d4c-9fe3-5fb3019e5bd0 service swift] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:01:07.006853 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a91555-c826-4d4c-9fe3-5fb3019e5bd0 service swift] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:01:07.007128 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a91555-c826-4d4c-9fe3-5fb3019e5bd0 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:01:07.015629 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a91555-c826-4d4c-9fe3-5fb3019e5bd0 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:01:07.017204 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a91555-c826-4d4c-9fe3-5fb3019e5bd0 service swift] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:01:07.025862 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1381/2771] 10.222.0.27 () {60 vars in 1384 bytes} [Sun Sep 11 01:01:06 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 11 01:01:07.253526 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23cca27f-47db-4788-bca6-4f08679690c8 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:01:07.253999 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23cca27f-47db-4788-bca6-4f08679690c8 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:01:07.254098 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23cca27f-47db-4788-bca6-4f08679690c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:01:07.274194 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-23cca27f-47db-4788-bca6-4f08679690c8 None None] MFA Rules not processed for user `9fc4274836844ae4ad5d16d74c36f671`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:01:07.275417 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-23cca27f-47db-4788-bca6-4f08679690c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:01:07.283657 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1391/2772] 10.222.0.27 () {58 vars in 1009 bytes} [Sun Sep 11 01:01:07 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 01:01:07.295997 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a91555-c826-4d4c-9fe3-5fb3019e5bd0 service swift] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:01:07.306767 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a91555-c826-4d4c-9fe3-5fb3019e5bd0 service swift] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:01:07.311941 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa9c202-d4b6-4822-9960-d779364a6f3d service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bcb73dec73464a3abc4b774abb4f416f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:01:07.312471 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa9c202-d4b6-4822-9960-d779364a6f3d service swift] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:01:07.312747 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa9c202-d4b6-4822-9960-d779364a6f3d service swift] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:01:07.313016 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa9c202-d4b6-4822-9960-d779364a6f3d service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:01:07.322198 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efa9c202-d4b6-4822-9960-d779364a6f3d service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:01:07.323859 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efa9c202-d4b6-4822-9960-d779364a6f3d service swift] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:01:07.332710 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1382/2773] 10.222.0.27 () {60 vars in 1384 bytes} [Sun Sep 11 01:01:07 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 11 01:01:07.658848 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c691f8e-dbcc-4262-bb23-cf9ab12ab79d None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:01:07.659285 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c691f8e-dbcc-4262-bb23-cf9ab12ab79d None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:01:07.659561 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c691f8e-dbcc-4262-bb23-cf9ab12ab79d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:01:07.680600 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-6c691f8e-dbcc-4262-bb23-cf9ab12ab79d None None] MFA Rules not processed for user `9fc4274836844ae4ad5d16d74c36f671`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:01:07.684701 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-6c691f8e-dbcc-4262-bb23-cf9ab12ab79d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:01:07.692402 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1392/2774] 10.222.0.27 () {58 vars in 967 bytes} [Sun Sep 11 01:01:07 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 01:01:07.706026 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa9c202-d4b6-4822-9960-d779364a6f3d service swift] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:01:07.717416 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa9c202-d4b6-4822-9960-d779364a6f3d service swift] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:01:07.723715 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35f0730-e008-4c12-98c8-8e7f59ddb869 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bcb73dec73464a3abc4b774abb4f416f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:01:07.724628 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35f0730-e008-4c12-98c8-8e7f59ddb869 service swift] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:01:07.725067 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35f0730-e008-4c12-98c8-8e7f59ddb869 service swift] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:01:07.725440 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35f0730-e008-4c12-98c8-8e7f59ddb869 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:01:07.738381 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b35f0730-e008-4c12-98c8-8e7f59ddb869 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:01:07.740782 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b35f0730-e008-4c12-98c8-8e7f59ddb869 service swift] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:01:07.749147 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1383/2775] 10.222.0.27 () {60 vars in 1384 bytes} [Sun Sep 11 01:01:07 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 11 01:01:09.863229 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c691f8e-dbcc-4262-bb23-cf9ab12ab79d None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:01:09.924590 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c691f8e-dbcc-4262-bb23-cf9ab12ab79d None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:01:09.929191 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b962edc-839b-4166-9939-812f89e69a18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:01:09.929811 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b962edc-839b-4166-9939-812f89e69a18 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:01:09.930088 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b962edc-839b-4166-9939-812f89e69a18 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:01:09.930365 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b962edc-839b-4166-9939-812f89e69a18 admin admin] PATH_INFO: `/v3/users/e0a15a4c53da4903a3f1444db955ab79` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:01:09.931501 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b962edc-839b-4166-9939-812f89e69a18 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e0a15a4c53da4903a3f1444db955ab79)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:01:09.933404 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b962edc-839b-4166-9939-812f89e69a18 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:01:09.979905 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-2b962edc-839b-4166-9939-812f89e69a18 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e0a15a4c53da4903a3f1444db955ab79'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:01:09.980553 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-2b962edc-839b-4166-9939-812f89e69a18 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e0a15a4c53da4903a3f1444db955ab79'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:01:09.981157 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-2b962edc-839b-4166-9939-812f89e69a18 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e0a15a4c53da4903a3f1444db955ab79'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:01:09.988042 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-2b962edc-839b-4166-9939-812f89e69a18 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e0a15a4c53da4903a3f1444db955ab79'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:01:09.999090 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=91,keystone:DELETE=9 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:01:10.000407 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-2b962edc-839b-4166-9939-812f89e69a18 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e0a15a4c53da4903a3f1444db955ab79'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:01:10.012941 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1393/2776] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 01:01:09 2022] DELETE /identity/v3/users/e0a15a4c53da4903a3f1444db955ab79 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 01:01:10.017667 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177e573c-0dc3-4ec4-95df-db7070aefd5e None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:01:10.018128 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177e573c-0dc3-4ec4-95df-db7070aefd5e None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:01:10.018344 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177e573c-0dc3-4ec4-95df-db7070aefd5e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:01:10.039307 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-177e573c-0dc3-4ec4-95df-db7070aefd5e None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:01:10.040518 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-177e573c-0dc3-4ec4-95df-db7070aefd5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:01:10.103657 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1384/2777] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 01:01:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 01:01:10.113509 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b962edc-839b-4166-9939-812f89e69a18 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:01:10.114969 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-2b962edc-839b-4166-9939-812f89e69a18 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:01:10.198637 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b962edc-839b-4166-9939-812f89e69a18 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:01:10.202220 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0398010-1ca3-44b8-b6be-9126c9a29314 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:01:10.202846 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0398010-1ca3-44b8-b6be-9126c9a29314 service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:01:10.203125 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0398010-1ca3-44b8-b6be-9126c9a29314 service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:01:10.203393 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0398010-1ca3-44b8-b6be-9126c9a29314 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:01:10.212901 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0398010-1ca3-44b8-b6be-9126c9a29314 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:01:10.214316 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0398010-1ca3-44b8-b6be-9126c9a29314 service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:01:10.224348 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1394/2778] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 01:01:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 01:01:10.399617 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-177e573c-0dc3-4ec4-95df-db7070aefd5e None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:01:10.401127 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-177e573c-0dc3-4ec4-95df-db7070aefd5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:01:10.413771 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-177e573c-0dc3-4ec4-95df-db7070aefd5e None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:01:10.420016 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5b57cf-6a24-4e54-a535-99bfce85fce1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:01:10.420900 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5b57cf-6a24-4e54-a535-99bfce85fce1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:01:10.421217 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5b57cf-6a24-4e54-a535-99bfce85fce1 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:01:10.421556 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5b57cf-6a24-4e54-a535-99bfce85fce1 admin admin] PATH_INFO: `/v3/projects/d99b9dcee07048b49f75a4e0172b42be` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:01:10.423495 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e5b57cf-6a24-4e54-a535-99bfce85fce1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d99b9dcee07048b49f75a4e0172b42be)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:01:10.425595 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e5b57cf-6a24-4e54-a535-99bfce85fce1 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:01:10.474548 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:01:10.477599 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-4e5b57cf-6a24-4e54-a535-99bfce85fce1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd99b9dcee07048b49f75a4e0172b42be', 'request_id': 'req-4e5b57cf-6a24-4e54-a535-99bfce85fce1'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:01:10.483891 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-4e5b57cf-6a24-4e54-a535-99bfce85fce1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd99b9dcee07048b49f75a4e0172b42be', 'request_id': 'req-4e5b57cf-6a24-4e54-a535-99bfce85fce1'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:01:10.489449 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-4e5b57cf-6a24-4e54-a535-99bfce85fce1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd99b9dcee07048b49f75a4e0172b42be', 'request_id': 'req-4e5b57cf-6a24-4e54-a535-99bfce85fce1'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:01:10.501887 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-4e5b57cf-6a24-4e54-a535-99bfce85fce1 admin admin] The token cache is being invalidate because project d99b9dcee07048b49f75a4e0172b42be was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87923) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 01:01:10.502275 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-4e5b57cf-6a24-4e54-a535-99bfce85fce1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:01:10.502924 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-4e5b57cf-6a24-4e54-a535-99bfce85fce1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:01:10.504393 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1385/2779] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 01:01:10 2022] DELETE /identity/v3/projects/d99b9dcee07048b49f75a4e0172b42be => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 01:01:10.510129 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5faa0397-4875-458b-b6c7-5dcf14ba034a None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:01:10.510415 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5faa0397-4875-458b-b6c7-5dcf14ba034a None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:01:10.510701 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5faa0397-4875-458b-b6c7-5dcf14ba034a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:01:10.531024 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-5faa0397-4875-458b-b6c7-5dcf14ba034a None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:01:10.532211 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-5faa0397-4875-458b-b6c7-5dcf14ba034a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:01:10.593080 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1395/2780] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 01:01:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 01:01:10.601795 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506270b8-3255-4961-892b-9ca227cc9a23 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:01:10.602072 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506270b8-3255-4961-892b-9ca227cc9a23 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:01:10.602344 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506270b8-3255-4961-892b-9ca227cc9a23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:01:10.622312 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-506270b8-3255-4961-892b-9ca227cc9a23 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:01:10.623524 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-506270b8-3255-4961-892b-9ca227cc9a23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:01:10.631819 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1386/2781] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 01:01:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 01:01:10.636146 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5faa0397-4875-458b-b6c7-5dcf14ba034a None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:01:10.645694 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5faa0397-4875-458b-b6c7-5dcf14ba034a None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:01:10.650515 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f3aebf0-aa52-4120-90d4-aaaf1d19675f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:01:10.651142 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3aebf0-aa52-4120-90d4-aaaf1d19675f admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:01:10.651420 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3aebf0-aa52-4120-90d4-aaaf1d19675f admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:01:10.651688 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3aebf0-aa52-4120-90d4-aaaf1d19675f admin admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:01:10.652328 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f3aebf0-aa52-4120-90d4-aaaf1d19675f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 01:01:10.652628 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f3aebf0-aa52-4120-90d4-aaaf1d19675f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:01:10.654203 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f3aebf0-aa52-4120-90d4-aaaf1d19675f admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:01:10.665977 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1396/2782] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 01:01:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 01:01:10.670300 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-506270b8-3255-4961-892b-9ca227cc9a23 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:01:10.680280 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-506270b8-3255-4961-892b-9ca227cc9a23 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:01:10.684954 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3577364-98fd-4788-b7dd-9f969555a7c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:01:10.685542 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3577364-98fd-4788-b7dd-9f969555a7c8 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:01:10.685819 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3577364-98fd-4788-b7dd-9f969555a7c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:01:10.686093 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3577364-98fd-4788-b7dd-9f969555a7c8 admin admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:01:10.686731 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3577364-98fd-4788-b7dd-9f969555a7c8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:01:10.688653 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3577364-98fd-4788-b7dd-9f969555a7c8 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:01:10.717342 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1387/2783] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 01:01:10 2022] POST /identity/v3/projects => generated 399 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 01:01:10.721767 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f3aebf0-aa52-4120-90d4-aaaf1d19675f admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:01:10.732152 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f3aebf0-aa52-4120-90d4-aaaf1d19675f admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:01:10.787722 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-654f4faf-fb8e-4fbf-ad32-8cd88cdc86fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:01:10.788330 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654f4faf-fb8e-4fbf-ad32-8cd88cdc86fa admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:01:10.788611 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654f4faf-fb8e-4fbf-ad32-8cd88cdc86fa admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:01:10.788885 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654f4faf-fb8e-4fbf-ad32-8cd88cdc86fa admin admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:01:10.789522 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-654f4faf-fb8e-4fbf-ad32-8cd88cdc86fa admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:01:10.791487 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-654f4faf-fb8e-4fbf-ad32-8cd88cdc86fa admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:01:10.799348 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=98,keystone:INSERT=1,keystone:DELETE=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:01:10.820356 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1397/2784] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 01:01:10 2022] POST /identity/v3/users => generated 300 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 01:01:10.821556 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3577364-98fd-4788-b7dd-9f969555a7c8 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:01:10.831961 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3577364-98fd-4788-b7dd-9f969555a7c8 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:01:10.836874 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d948ebfa-0cf5-4292-8fa8-9488685d7621 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:01:10.837519 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d948ebfa-0cf5-4292-8fa8-9488685d7621 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:01:10.837800 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d948ebfa-0cf5-4292-8fa8-9488685d7621 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:01:10.838069 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d948ebfa-0cf5-4292-8fa8-9488685d7621 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:01:10.838743 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d948ebfa-0cf5-4292-8fa8-9488685d7621 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:01:10.840410 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d948ebfa-0cf5-4292-8fa8-9488685d7621 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:01:10.850773 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1388/2785] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 01:01:10 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 01:01:10.855031 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-654f4faf-fb8e-4fbf-ad32-8cd88cdc86fa admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:01:10.866027 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-654f4faf-fb8e-4fbf-ad32-8cd88cdc86fa admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:01:10.872217 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d6d35a-e108-4afc-a4ec-4b288e836ddb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:01:10.872813 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d6d35a-e108-4afc-a4ec-4b288e836ddb admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:01:10.873091 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d6d35a-e108-4afc-a4ec-4b288e836ddb admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:01:10.873361 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d6d35a-e108-4afc-a4ec-4b288e836ddb admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:01:10.874000 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8d6d35a-e108-4afc-a4ec-4b288e836ddb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:01:10.875626 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8d6d35a-e108-4afc-a4ec-4b288e836ddb admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:01:10.887941 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1398/2786] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 01:01:10 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 01:01:10.892792 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d948ebfa-0cf5-4292-8fa8-9488685d7621 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:01:10.903396 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d948ebfa-0cf5-4292-8fa8-9488685d7621 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:01:10.908530 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf4326aa-d0ba-4b74-a777-a51e46840750 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:01:10.909177 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4326aa-d0ba-4b74-a777-a51e46840750 admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:01:10.909479 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4326aa-d0ba-4b74-a777-a51e46840750 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:01:10.909748 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4326aa-d0ba-4b74-a777-a51e46840750 admin admin] PATH_INFO: `/v3/projects/90c4d0d118634af08eadb360d0abd817/users/8ba24604d2d74d6e8b2cfe603b8cd6ba/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:01:10.925795 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf4326aa-d0ba-4b74-a777-a51e46840750 admin admin] RBAC: Authorizing `identity:create_grant(project_id=90c4d0d118634af08eadb360d0abd817, user_id=8ba24604d2d74d6e8b2cfe603b8cd6ba, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:01:10.929165 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf4326aa-d0ba-4b74-a777-a51e46840750 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:01:10.944106 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1389/2787] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 01:01:10 2022] PUT /identity/v3/projects/90c4d0d118634af08eadb360d0abd817/users/8ba24604d2d74d6e8b2cfe603b8cd6ba/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 01:01:10.954009 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b8ecc9-14a7-428d-84b8-63d14fbb50ba None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:01:10.954295 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b8ecc9-14a7-428d-84b8-63d14fbb50ba None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:01:10.954592 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b8ecc9-14a7-428d-84b8-63d14fbb50ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:01:10.987685 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-f7b8ecc9-14a7-428d-84b8-63d14fbb50ba None None] MFA Rules not processed for user `8ba24604d2d74d6e8b2cfe603b8cd6ba`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:01:10.988798 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-f7b8ecc9-14a7-428d-84b8-63d14fbb50ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:01:11.053306 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1399/2788] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 01:01:10 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 01:01:11.064763 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf4326aa-d0ba-4b74-a777-a51e46840750 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:01:11.066474 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-bf4326aa-d0ba-4b74-a777-a51e46840750 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:01:11.144986 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf4326aa-d0ba-4b74-a777-a51e46840750 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:01:11.148827 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ee0af6d-49bc-4eca-a5bb-4aff84de861d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:01:11.149417 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee0af6d-49bc-4eca-a5bb-4aff84de861d service cinder] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:01:11.149692 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee0af6d-49bc-4eca-a5bb-4aff84de861d service cinder] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:01:11.149959 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee0af6d-49bc-4eca-a5bb-4aff84de861d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:01:11.158451 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ee0af6d-49bc-4eca-a5bb-4aff84de861d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:01:11.159934 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ee0af6d-49bc-4eca-a5bb-4aff84de861d service cinder] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:01:11.169490 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1390/2789] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 01:01:11 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 01:01:11.730507 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b8ecc9-14a7-428d-84b8-63d14fbb50ba None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:01:11.732013 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-f7b8ecc9-14a7-428d-84b8-63d14fbb50ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:01:11.798129 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b8ecc9-14a7-428d-84b8-63d14fbb50ba None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:01:11.803658 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3529d65e-d3bd-4e59-a56b-2f7949ccb0f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:01:11.804285 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3529d65e-d3bd-4e59-a56b-2f7949ccb0f5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:01:11.804558 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3529d65e-d3bd-4e59-a56b-2f7949ccb0f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:01:11.804826 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3529d65e-d3bd-4e59-a56b-2f7949ccb0f5 admin admin] PATH_INFO: `/v3/users/78a2199d3f1d4bc5b1f988ae06c0d609` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:01:11.805958 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3529d65e-d3bd-4e59-a56b-2f7949ccb0f5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=78a2199d3f1d4bc5b1f988ae06c0d609)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:01:11.807941 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3529d65e-d3bd-4e59-a56b-2f7949ccb0f5 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:01:11.853474 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-3529d65e-d3bd-4e59-a56b-2f7949ccb0f5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '78a2199d3f1d4bc5b1f988ae06c0d609'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:01:11.854135 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-3529d65e-d3bd-4e59-a56b-2f7949ccb0f5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '78a2199d3f1d4bc5b1f988ae06c0d609'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:01:11.854726 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-3529d65e-d3bd-4e59-a56b-2f7949ccb0f5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '78a2199d3f1d4bc5b1f988ae06c0d609'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:01:11.861702 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-3529d65e-d3bd-4e59-a56b-2f7949ccb0f5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '78a2199d3f1d4bc5b1f988ae06c0d609'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:01:11.872239 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:INSERT=3,keystone:SELECT=89,keystone:DELETE=8 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:01:11.874755 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-3529d65e-d3bd-4e59-a56b-2f7949ccb0f5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '78a2199d3f1d4bc5b1f988ae06c0d609'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:01:11.888662 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1400/2790] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 01:01:11 2022] DELETE /identity/v3/users/78a2199d3f1d4bc5b1f988ae06c0d609 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 01:01:12.111380 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ee0af6d-49bc-4eca-a5bb-4aff84de861d service cinder] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:01:12.123125 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ee0af6d-49bc-4eca-a5bb-4aff84de861d service cinder] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:01:12.177242 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:DELETE=4,keystone:SELECT=93,keystone:INSERT=3 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:01:12.188195 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a235fde-a2a2-4be6-881d-7fdcaccbceaa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:01:12.189062 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a235fde-a2a2-4be6-881d-7fdcaccbceaa admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:01:12.189337 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a235fde-a2a2-4be6-881d-7fdcaccbceaa admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:01:12.189607 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a235fde-a2a2-4be6-881d-7fdcaccbceaa admin admin] PATH_INFO: `/v3/projects/de36b740b8234aca8317d5e4440decec` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:01:12.190719 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a235fde-a2a2-4be6-881d-7fdcaccbceaa admin admin] RBAC: Authorizing `identity:delete_project(project_id=de36b740b8234aca8317d5e4440decec)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:01:12.193678 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a235fde-a2a2-4be6-881d-7fdcaccbceaa admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:01:12.238616 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-8a235fde-a2a2-4be6-881d-7fdcaccbceaa admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'de36b740b8234aca8317d5e4440decec', 'request_id': 'req-8a235fde-a2a2-4be6-881d-7fdcaccbceaa'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:01:12.245267 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-8a235fde-a2a2-4be6-881d-7fdcaccbceaa admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'de36b740b8234aca8317d5e4440decec', 'request_id': 'req-8a235fde-a2a2-4be6-881d-7fdcaccbceaa'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:01:12.250857 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-8a235fde-a2a2-4be6-881d-7fdcaccbceaa admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'de36b740b8234aca8317d5e4440decec', 'request_id': 'req-8a235fde-a2a2-4be6-881d-7fdcaccbceaa'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:01:12.263538 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-8a235fde-a2a2-4be6-881d-7fdcaccbceaa admin admin] The token cache is being invalidate because project de36b740b8234aca8317d5e4440decec was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87923) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 01:01:12.263824 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-8a235fde-a2a2-4be6-881d-7fdcaccbceaa admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:01:12.264433 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-8a235fde-a2a2-4be6-881d-7fdcaccbceaa admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:01:12.265915 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1391/2791] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 01:01:12 2022] DELETE /identity/v3/projects/de36b740b8234aca8317d5e4440decec => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 01:01:19.265743 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3529d65e-d3bd-4e59-a56b-2f7949ccb0f5 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:01:19.267188 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-3529d65e-d3bd-4e59-a56b-2f7949ccb0f5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:01:19.281084 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3529d65e-d3bd-4e59-a56b-2f7949ccb0f5 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:01:19.338913 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e42d5b02-1e01-469e-9a1e-2abdeae7880e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:01:19.339539 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e42d5b02-1e01-469e-9a1e-2abdeae7880e admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:01:19.339820 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e42d5b02-1e01-469e-9a1e-2abdeae7880e admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:01:19.340095 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e42d5b02-1e01-469e-9a1e-2abdeae7880e admin admin] PATH_INFO: `/v3/users/8ba24604d2d74d6e8b2cfe603b8cd6ba` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:01:19.341233 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e42d5b02-1e01-469e-9a1e-2abdeae7880e admin admin] RBAC: Authorizing `identity:delete_user(user_id=8ba24604d2d74d6e8b2cfe603b8cd6ba)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:01:19.343188 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e42d5b02-1e01-469e-9a1e-2abdeae7880e admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:01:19.388717 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-e42d5b02-1e01-469e-9a1e-2abdeae7880e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8ba24604d2d74d6e8b2cfe603b8cd6ba'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:01:19.389354 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-e42d5b02-1e01-469e-9a1e-2abdeae7880e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8ba24604d2d74d6e8b2cfe603b8cd6ba'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:01:19.389942 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-e42d5b02-1e01-469e-9a1e-2abdeae7880e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8ba24604d2d74d6e8b2cfe603b8cd6ba'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:01:19.397686 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-e42d5b02-1e01-469e-9a1e-2abdeae7880e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8ba24604d2d74d6e8b2cfe603b8cd6ba'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:01:19.409621 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-e42d5b02-1e01-469e-9a1e-2abdeae7880e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8ba24604d2d74d6e8b2cfe603b8cd6ba'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:01:19.422256 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1401/2792] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 01:01:19 2022] DELETE /identity/v3/users/8ba24604d2d74d6e8b2cfe603b8cd6ba => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 01:01:19.426899 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4dad2b-f049-4948-a53a-0016f6ebfe8d None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:01:19.427452 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4dad2b-f049-4948-a53a-0016f6ebfe8d None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:01:19.427452 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4dad2b-f049-4948-a53a-0016f6ebfe8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:01:19.446956 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-0b4dad2b-f049-4948-a53a-0016f6ebfe8d None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:01:19.448145 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-0b4dad2b-f049-4948-a53a-0016f6ebfe8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:01:19.506501 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1392/2793] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 01:01:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 01:01:19.516057 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e42d5b02-1e01-469e-9a1e-2abdeae7880e admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:01:19.517464 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-e42d5b02-1e01-469e-9a1e-2abdeae7880e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:01:19.574419 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e42d5b02-1e01-469e-9a1e-2abdeae7880e admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:01:19.578235 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4ce27fe-c505-47b7-839e-6c143fa023be service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:01:19.578886 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ce27fe-c505-47b7-839e-6c143fa023be service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:01:19.579168 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ce27fe-c505-47b7-839e-6c143fa023be service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:01:19.579445 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ce27fe-c505-47b7-839e-6c143fa023be service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:01:19.589306 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4ce27fe-c505-47b7-839e-6c143fa023be service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:01:19.590740 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4ce27fe-c505-47b7-839e-6c143fa023be service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:01:19.600129 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1402/2794] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 01:01:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 01:01:19.867469 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4dad2b-f049-4948-a53a-0016f6ebfe8d None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:01:19.869188 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-0b4dad2b-f049-4948-a53a-0016f6ebfe8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:01:19.882712 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4dad2b-f049-4948-a53a-0016f6ebfe8d None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:01:19.887600 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f4dd26-752e-417e-a61e-f9299ce0db50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:01:19.888310 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f4dd26-752e-417e-a61e-f9299ce0db50 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:01:19.888682 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f4dd26-752e-417e-a61e-f9299ce0db50 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:01:19.889026 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f4dd26-752e-417e-a61e-f9299ce0db50 admin admin] PATH_INFO: `/v3/projects/90c4d0d118634af08eadb360d0abd817` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:01:19.890219 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12f4dd26-752e-417e-a61e-f9299ce0db50 admin admin] RBAC: Authorizing `identity:delete_project(project_id=90c4d0d118634af08eadb360d0abd817)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:01:19.892236 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12f4dd26-752e-417e-a61e-f9299ce0db50 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:01:19.943610 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-12f4dd26-752e-417e-a61e-f9299ce0db50 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '90c4d0d118634af08eadb360d0abd817', 'request_id': 'req-12f4dd26-752e-417e-a61e-f9299ce0db50'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:01:19.949795 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-12f4dd26-752e-417e-a61e-f9299ce0db50 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '90c4d0d118634af08eadb360d0abd817', 'request_id': 'req-12f4dd26-752e-417e-a61e-f9299ce0db50'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:01:19.955655 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-12f4dd26-752e-417e-a61e-f9299ce0db50 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '90c4d0d118634af08eadb360d0abd817', 'request_id': 'req-12f4dd26-752e-417e-a61e-f9299ce0db50'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:01:19.967756 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-12f4dd26-752e-417e-a61e-f9299ce0db50 admin admin] The token cache is being invalidate because project 90c4d0d118634af08eadb360d0abd817 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87923) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 01:01:19.968045 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-12f4dd26-752e-417e-a61e-f9299ce0db50 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:01:19.968684 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-12f4dd26-752e-417e-a61e-f9299ce0db50 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:01:19.970177 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1393/2795] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 01:01:19 2022] DELETE /identity/v3/projects/90c4d0d118634af08eadb360d0abd817 => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 01:01:29.596045 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:DELETE=12,keystone:SELECT=82,keystone:INSERT=2 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:01:29.964687 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=81,keystone:DELETE=14,keystone:INSERT=2 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:02:37.162122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4ce27fe-c505-47b7-839e-6c143fa023be service neutron] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:02:37.172234 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4ce27fe-c505-47b7-839e-6c143fa023be service neutron] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:02:37.222040 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ecb288-f066-4f08-b2ff-86eac8d418cf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:02:37.222641 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ecb288-f066-4f08-b2ff-86eac8d418cf service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:02:37.222944 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ecb288-f066-4f08-b2ff-86eac8d418cf service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:02:37.223214 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ecb288-f066-4f08-b2ff-86eac8d418cf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:02:37.224912 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-21ecb288-f066-4f08-b2ff-86eac8d418cf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:02:37.308144 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21ecb288-f066-4f08-b2ff-86eac8d418cf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:02:37.309566 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21ecb288-f066-4f08-b2ff-86eac8d418cf service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:02:37.329365 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1403/2796] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 01:02:37 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 168 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 01:02:38.510513 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f4dd26-752e-417e-a61e-f9299ce0db50 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:02:38.511983 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-12f4dd26-752e-417e-a61e-f9299ce0db50 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:02:38.524105 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f4dd26-752e-417e-a61e-f9299ce0db50 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:02:38.527725 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7fccebe-f55f-4863-b635-144068380192 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:02:38.528296 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7fccebe-f55f-4863-b635-144068380192 service neutron] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:02:38.528578 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7fccebe-f55f-4863-b635-144068380192 service neutron] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:02:38.528850 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7fccebe-f55f-4863-b635-144068380192 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:02:38.530829 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-c7fccebe-f55f-4863-b635-144068380192 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:02:38.544021 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7fccebe-f55f-4863-b635-144068380192 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:02:38.545491 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7fccebe-f55f-4863-b635-144068380192 service neutron] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:02:38.557500 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1394/2797] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 01:02:38 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 01:02:47.326765 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=63 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:02:48.551563 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=6 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:03:00.724087 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ecb288-f066-4f08-b2ff-86eac8d418cf service neutron] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:03:00.725505 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-21ecb288-f066-4f08-b2ff-86eac8d418cf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:03:00.736889 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ecb288-f066-4f08-b2ff-86eac8d418cf service neutron] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:03:00.740527 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3cbe9fb-af2a-4f3a-9bce-0cf01a4355cf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:03:00.741122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3cbe9fb-af2a-4f3a-9bce-0cf01a4355cf service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:03:00.741400 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3cbe9fb-af2a-4f3a-9bce-0cf01a4355cf service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:03:00.741670 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3cbe9fb-af2a-4f3a-9bce-0cf01a4355cf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:03:00.743355 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-e3cbe9fb-af2a-4f3a-9bce-0cf01a4355cf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:03:00.752686 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3cbe9fb-af2a-4f3a-9bce-0cf01a4355cf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:03:00.754106 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3cbe9fb-af2a-4f3a-9bce-0cf01a4355cf service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:03:00.762075 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1404/2798] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 01:03:00 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 01:03:05.499415 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7fccebe-f55f-4863-b635-144068380192 service neutron] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:03:05.508327 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7fccebe-f55f-4863-b635-144068380192 service neutron] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:03:05.512282 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcf6a927-c14f-471f-8364-401c7673d0f0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:03:05.512883 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf6a927-c14f-471f-8364-401c7673d0f0 service neutron] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:03:05.513162 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf6a927-c14f-471f-8364-401c7673d0f0 service neutron] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:03:05.513430 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf6a927-c14f-471f-8364-401c7673d0f0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:03:05.515229 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-fcf6a927-c14f-471f-8364-401c7673d0f0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:03:05.579129 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcf6a927-c14f-471f-8364-401c7673d0f0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:03:05.580781 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcf6a927-c14f-471f-8364-401c7673d0f0 service neutron] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:03:05.591007 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1395/2799] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 01:03:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 01:03:10.758408 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=6 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:03:11.627329 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3cbe9fb-af2a-4f3a-9bce-0cf01a4355cf service neutron] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:03:11.628773 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-e3cbe9fb-af2a-4f3a-9bce-0cf01a4355cf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:03:11.644360 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3cbe9fb-af2a-4f3a-9bce-0cf01a4355cf service neutron] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:03:11.648954 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8edcaccc-a315-4cfe-94c6-f3b45217cb99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:03:11.649577 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8edcaccc-a315-4cfe-94c6-f3b45217cb99 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:03:11.649859 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8edcaccc-a315-4cfe-94c6-f3b45217cb99 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:03:11.650161 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8edcaccc-a315-4cfe-94c6-f3b45217cb99 admin admin] PATH_INFO: `/v3/users/e1012f856881485cb40d916131a5d763` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:03:11.651894 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8edcaccc-a315-4cfe-94c6-f3b45217cb99 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e1012f856881485cb40d916131a5d763)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:03:11.653827 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8edcaccc-a315-4cfe-94c6-f3b45217cb99 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:03:11.701522 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-8edcaccc-a315-4cfe-94c6-f3b45217cb99 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e1012f856881485cb40d916131a5d763'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:03:11.703231 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-8edcaccc-a315-4cfe-94c6-f3b45217cb99 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e1012f856881485cb40d916131a5d763'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:03:11.703927 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-8edcaccc-a315-4cfe-94c6-f3b45217cb99 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e1012f856881485cb40d916131a5d763'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:03:11.711156 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-8edcaccc-a315-4cfe-94c6-f3b45217cb99 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e1012f856881485cb40d916131a5d763'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:03:11.724646 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-8edcaccc-a315-4cfe-94c6-f3b45217cb99 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e1012f856881485cb40d916131a5d763'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:03:11.738155 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1405/2800] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 01:03:11 2022] DELETE /identity/v3/users/e1012f856881485cb40d916131a5d763 => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 01:03:11.914328 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcf6a927-c14f-471f-8364-401c7673d0f0 service neutron] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:03:11.924906 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcf6a927-c14f-471f-8364-401c7673d0f0 service neutron] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:03:11.982219 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f530c9b-510a-4f87-95a1-758ba7eee9b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:03:11.982874 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f530c9b-510a-4f87-95a1-758ba7eee9b8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:03:11.983152 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f530c9b-510a-4f87-95a1-758ba7eee9b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:03:11.983420 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f530c9b-510a-4f87-95a1-758ba7eee9b8 admin admin] PATH_INFO: `/v3/projects/a1368870ddbd49be857e9ac128bc684e` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:03:11.984514 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f530c9b-510a-4f87-95a1-758ba7eee9b8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a1368870ddbd49be857e9ac128bc684e)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:03:11.986415 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f530c9b-510a-4f87-95a1-758ba7eee9b8 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:03:12.032437 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-6f530c9b-510a-4f87-95a1-758ba7eee9b8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a1368870ddbd49be857e9ac128bc684e', 'request_id': 'req-6f530c9b-510a-4f87-95a1-758ba7eee9b8'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:03:12.038866 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-6f530c9b-510a-4f87-95a1-758ba7eee9b8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a1368870ddbd49be857e9ac128bc684e', 'request_id': 'req-6f530c9b-510a-4f87-95a1-758ba7eee9b8'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:03:12.044799 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-6f530c9b-510a-4f87-95a1-758ba7eee9b8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a1368870ddbd49be857e9ac128bc684e', 'request_id': 'req-6f530c9b-510a-4f87-95a1-758ba7eee9b8'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:03:12.057685 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-6f530c9b-510a-4f87-95a1-758ba7eee9b8 admin admin] The token cache is being invalidate because project a1368870ddbd49be857e9ac128bc684e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87923) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 01:03:12.057899 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-6f530c9b-510a-4f87-95a1-758ba7eee9b8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:03:12.058524 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-6f530c9b-510a-4f87-95a1-758ba7eee9b8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:03:12.060021 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1396/2801] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 01:03:11 2022] DELETE /identity/v3/projects/a1368870ddbd49be857e9ac128bc684e => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 01:03:12.063989 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8edcaccc-a315-4cfe-94c6-f3b45217cb99 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:03:12.065415 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-8edcaccc-a315-4cfe-94c6-f3b45217cb99 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:03:12.079489 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8edcaccc-a315-4cfe-94c6-f3b45217cb99 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:03:12.135960 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccf62eab-586f-44fd-a120-f0addbd044de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:03:12.136580 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf62eab-586f-44fd-a120-f0addbd044de admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:03:12.136858 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf62eab-586f-44fd-a120-f0addbd044de admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:03:12.137129 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf62eab-586f-44fd-a120-f0addbd044de admin admin] PATH_INFO: `/v3/users/39a914ebb07648fdb5cfd683da419e11` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:03:12.138254 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccf62eab-586f-44fd-a120-f0addbd044de admin admin] RBAC: Authorizing `identity:delete_user(user_id=39a914ebb07648fdb5cfd683da419e11)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:03:12.140197 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccf62eab-586f-44fd-a120-f0addbd044de admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:03:12.186246 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-ccf62eab-586f-44fd-a120-f0addbd044de admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '39a914ebb07648fdb5cfd683da419e11'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:03:12.186904 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-ccf62eab-586f-44fd-a120-f0addbd044de admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '39a914ebb07648fdb5cfd683da419e11'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:03:12.187501 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-ccf62eab-586f-44fd-a120-f0addbd044de admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '39a914ebb07648fdb5cfd683da419e11'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:03:12.194301 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-ccf62eab-586f-44fd-a120-f0addbd044de admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '39a914ebb07648fdb5cfd683da419e11'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:03:12.205760 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-ccf62eab-586f-44fd-a120-f0addbd044de admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '39a914ebb07648fdb5cfd683da419e11'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:03:12.207370 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=80,keystone:DELETE=19,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:03:12.220833 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1406/2802] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 01:03:12 2022] DELETE /identity/v3/users/39a914ebb07648fdb5cfd683da419e11 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 01:03:12.397645 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f530c9b-510a-4f87-95a1-758ba7eee9b8 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:03:12.399144 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-6f530c9b-510a-4f87-95a1-758ba7eee9b8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:03:12.418450 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f530c9b-510a-4f87-95a1-758ba7eee9b8 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:03:12.443327 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:03:12.481539 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22977553-89db-46c6-a77a-c701bcd7d434 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:03:12.482163 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22977553-89db-46c6-a77a-c701bcd7d434 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:03:12.482436 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22977553-89db-46c6-a77a-c701bcd7d434 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:03:12.482705 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22977553-89db-46c6-a77a-c701bcd7d434 admin admin] PATH_INFO: `/v3/projects/1ef38a4efe0a41548e5ae4a9d7d414ea` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:03:12.483841 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22977553-89db-46c6-a77a-c701bcd7d434 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1ef38a4efe0a41548e5ae4a9d7d414ea)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:03:12.485727 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22977553-89db-46c6-a77a-c701bcd7d434 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:03:12.531203 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-22977553-89db-46c6-a77a-c701bcd7d434 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1ef38a4efe0a41548e5ae4a9d7d414ea', 'request_id': 'req-22977553-89db-46c6-a77a-c701bcd7d434'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:03:12.537149 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-22977553-89db-46c6-a77a-c701bcd7d434 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1ef38a4efe0a41548e5ae4a9d7d414ea', 'request_id': 'req-22977553-89db-46c6-a77a-c701bcd7d434'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:03:12.542690 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-22977553-89db-46c6-a77a-c701bcd7d434 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1ef38a4efe0a41548e5ae4a9d7d414ea', 'request_id': 'req-22977553-89db-46c6-a77a-c701bcd7d434'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:03:12.554391 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-22977553-89db-46c6-a77a-c701bcd7d434 admin admin] The token cache is being invalidate because project 1ef38a4efe0a41548e5ae4a9d7d414ea was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87923) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 01:03:12.554672 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-22977553-89db-46c6-a77a-c701bcd7d434 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:03:12.555359 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-22977553-89db-46c6-a77a-c701bcd7d434 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:03:12.556918 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1397/2803] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 01:03:12 2022] DELETE /identity/v3/projects/1ef38a4efe0a41548e5ae4a9d7d414ea => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 01:03:20.378196 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccf62eab-586f-44fd-a120-f0addbd044de admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:03:20.379747 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-ccf62eab-586f-44fd-a120-f0addbd044de admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:03:20.460072 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccf62eab-586f-44fd-a120-f0addbd044de admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:03:20.539746 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-460afc1a-faed-475f-86a0-9881d056e167 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '835def0e6c6d4365a2767051509eb0d6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'ResellerAdmin', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:03:20.540563 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460afc1a-faed-475f-86a0-9881d056e167 service nova] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:03:20.540728 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460afc1a-faed-475f-86a0-9881d056e167 service nova] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:03:20.542179 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460afc1a-faed-475f-86a0-9881d056e167 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:03:20.544255 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-460afc1a-faed-475f-86a0-9881d056e167 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:03:20.557490 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-460afc1a-faed-475f-86a0-9881d056e167 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:03:20.559447 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-460afc1a-faed-475f-86a0-9881d056e167 service nova] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:03:20.571051 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1407/2804] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 01:03:20 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 193 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 01:03:22.551434 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=40,keystone:DELETE=7,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:03:23.799252 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822af2e8-531f-4567-b6a7-729a408cb640 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:03:23.799854 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822af2e8-531f-4567-b6a7-729a408cb640 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:03:23.799854 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822af2e8-531f-4567-b6a7-729a408cb640 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:03:23.819659 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-822af2e8-531f-4567-b6a7-729a408cb640 None None] MFA Rules not processed for user `58b3528fd06b4d438caa92fd88383ed5`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:03:23.820916 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-822af2e8-531f-4567-b6a7-729a408cb640 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:03:23.875196 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1398/2805] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 01:03:23 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 01:03:24.495683 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-460afc1a-faed-475f-86a0-9881d056e167 service nova] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:03:24.497112 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-460afc1a-faed-475f-86a0-9881d056e167 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:03:24.567795 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-460afc1a-faed-475f-86a0-9881d056e167 service nova] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:03:24.572801 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f13459e-d8ec-4c46-be00-431c99745c28 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cdbfc2ccd73e4a5eb8c999be5b094619', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:03:24.573401 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f13459e-d8ec-4c46-be00-431c99745c28 service placement] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:03:24.573701 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f13459e-d8ec-4c46-be00-431c99745c28 service placement] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:03:24.573970 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f13459e-d8ec-4c46-be00-431c99745c28 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:03:24.575700 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-1f13459e-d8ec-4c46-be00-431c99745c28 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:03:24.586985 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f13459e-d8ec-4c46-be00-431c99745c28 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:03:24.588823 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f13459e-d8ec-4c46-be00-431c99745c28 service placement] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:03:24.599831 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1408/2806] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 01:03:24 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 01:03:32.723943 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-822af2e8-531f-4567-b6a7-729a408cb640 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:03:32.725426 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-822af2e8-531f-4567-b6a7-729a408cb640 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:03:32.743483 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-822af2e8-531f-4567-b6a7-729a408cb640 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:03:32.750120 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb891a6a-ae11-445e-99d3-7822eae16a3c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '835def0e6c6d4365a2767051509eb0d6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'ResellerAdmin', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:03:32.751140 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb891a6a-ae11-445e-99d3-7822eae16a3c service nova] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:03:32.751655 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb891a6a-ae11-445e-99d3-7822eae16a3c service nova] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:03:32.752159 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb891a6a-ae11-445e-99d3-7822eae16a3c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:03:32.754235 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-cb891a6a-ae11-445e-99d3-7822eae16a3c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:03:32.771655 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb891a6a-ae11-445e-99d3-7822eae16a3c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:03:32.773309 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb891a6a-ae11-445e-99d3-7822eae16a3c service nova] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:03:32.787282 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1399/2807] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 01:03:32 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 64 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 01:03:34.594547 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:INSERT=1,keystone:DELETE=1,keystone:SELECT=96 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:03:42.780602 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=31 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:04:41.676775 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f13459e-d8ec-4c46-be00-431c99745c28 service placement] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:04:41.688295 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f13459e-d8ec-4c46-be00-431c99745c28 service placement] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:04:41.694280 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c448d78-e1bb-4153-896b-639154b02f6a service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cdbfc2ccd73e4a5eb8c999be5b094619', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:04:41.695229 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c448d78-e1bb-4153-896b-639154b02f6a service placement] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:41.695730 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c448d78-e1bb-4153-896b-639154b02f6a service placement] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:41.696227 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c448d78-e1bb-4153-896b-639154b02f6a service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:41.698211 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-1c448d78-e1bb-4153-896b-639154b02f6a service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:04:41.712219 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c448d78-e1bb-4153-896b-639154b02f6a service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:04:41.713823 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c448d78-e1bb-4153-896b-639154b02f6a service placement] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:04:41.735928 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1409/2808] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 01:04:41 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 60 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 01:04:41.756048 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb891a6a-ae11-445e-99d3-7822eae16a3c service nova] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:04:41.757478 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-cb891a6a-ae11-445e-99d3-7822eae16a3c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:04:41.827670 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb891a6a-ae11-445e-99d3-7822eae16a3c service nova] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:04:41.832347 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb0d138-5d23-4119-9e97-0d42ad97b9bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:04:41.832918 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb0d138-5d23-4119-9e97-0d42ad97b9bb admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:41.833199 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb0d138-5d23-4119-9e97-0d42ad97b9bb admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:41.833474 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb0d138-5d23-4119-9e97-0d42ad97b9bb admin admin] PATH_INFO: `/v3/users/58b3528fd06b4d438caa92fd88383ed5` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:41.834605 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcb0d138-5d23-4119-9e97-0d42ad97b9bb admin admin] RBAC: Authorizing `identity:delete_user(user_id=58b3528fd06b4d438caa92fd88383ed5)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:04:41.837642 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcb0d138-5d23-4119-9e97-0d42ad97b9bb admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:04:41.883661 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-dcb0d138-5d23-4119-9e97-0d42ad97b9bb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '58b3528fd06b4d438caa92fd88383ed5'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:04:41.884312 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-dcb0d138-5d23-4119-9e97-0d42ad97b9bb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '58b3528fd06b4d438caa92fd88383ed5'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:04:41.884916 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-dcb0d138-5d23-4119-9e97-0d42ad97b9bb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '58b3528fd06b4d438caa92fd88383ed5'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:04:41.892585 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-dcb0d138-5d23-4119-9e97-0d42ad97b9bb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '58b3528fd06b4d438caa92fd88383ed5'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:04:41.904831 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-dcb0d138-5d23-4119-9e97-0d42ad97b9bb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '58b3528fd06b4d438caa92fd88383ed5'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:04:41.917388 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1400/2809] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 01:04:41 2022] DELETE /identity/v3/users/58b3528fd06b4d438caa92fd88383ed5 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 01:04:42.087185 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c448d78-e1bb-4153-896b-639154b02f6a service placement] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:04:42.098029 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c448d78-e1bb-4153-896b-639154b02f6a service placement] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:04:42.158110 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edecff37-3430-4f52-abf4-f87575e5599c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:04:42.158739 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edecff37-3430-4f52-abf4-f87575e5599c admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:42.159046 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edecff37-3430-4f52-abf4-f87575e5599c admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:42.159319 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edecff37-3430-4f52-abf4-f87575e5599c admin admin] PATH_INFO: `/v3/projects/15810c36aaa44c5fbc87197f873a6260` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:42.160434 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edecff37-3430-4f52-abf4-f87575e5599c admin admin] RBAC: Authorizing `identity:delete_project(project_id=15810c36aaa44c5fbc87197f873a6260)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:04:42.162331 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edecff37-3430-4f52-abf4-f87575e5599c admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:04:42.215712 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-edecff37-3430-4f52-abf4-f87575e5599c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '15810c36aaa44c5fbc87197f873a6260', 'request_id': 'req-edecff37-3430-4f52-abf4-f87575e5599c'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:04:42.222177 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-edecff37-3430-4f52-abf4-f87575e5599c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '15810c36aaa44c5fbc87197f873a6260', 'request_id': 'req-edecff37-3430-4f52-abf4-f87575e5599c'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:04:42.229773 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-edecff37-3430-4f52-abf4-f87575e5599c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '15810c36aaa44c5fbc87197f873a6260', 'request_id': 'req-edecff37-3430-4f52-abf4-f87575e5599c'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:04:42.240980 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-edecff37-3430-4f52-abf4-f87575e5599c admin admin] The token cache is being invalidate because project 15810c36aaa44c5fbc87197f873a6260 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87924) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 01:04:42.241271 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-edecff37-3430-4f52-abf4-f87575e5599c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:04:42.241895 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-edecff37-3430-4f52-abf4-f87575e5599c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:04:42.243430 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1410/2810] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 01:04:42 2022] DELETE /identity/v3/projects/15810c36aaa44c5fbc87197f873a6260 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 01:04:42.248420 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ebaa48a-445c-4448-94ab-4def67e42afe None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:42.248708 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ebaa48a-445c-4448-94ab-4def67e42afe None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:42.248982 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ebaa48a-445c-4448-94ab-4def67e42afe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:42.268312 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-5ebaa48a-445c-4448-94ab-4def67e42afe None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:04:42.269541 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-5ebaa48a-445c-4448-94ab-4def67e42afe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:04:42.328268 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1401/2811] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 01:04:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 01:04:42.336132 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d7566d-2c88-43e1-9a5d-0e6a08ca74c8 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:42.336418 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d7566d-2c88-43e1-9a5d-0e6a08ca74c8 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:42.336689 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d7566d-2c88-43e1-9a5d-0e6a08ca74c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:42.357486 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-43d7566d-2c88-43e1-9a5d-0e6a08ca74c8 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:04:42.358754 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-43d7566d-2c88-43e1-9a5d-0e6a08ca74c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:04:42.366693 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1411/2812] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 01:04:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 01:04:42.370998 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ebaa48a-445c-4448-94ab-4def67e42afe None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:04:42.381513 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ebaa48a-445c-4448-94ab-4def67e42afe None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:04:42.386045 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb652e2-0c6a-4a08-8397-745b1c945660 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:04:42.386638 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb652e2-0c6a-4a08-8397-745b1c945660 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:42.387755 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb652e2-0c6a-4a08-8397-745b1c945660 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:42.388080 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb652e2-0c6a-4a08-8397-745b1c945660 admin admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:42.388869 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb652e2-0c6a-4a08-8397-745b1c945660 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 01:04:42.389171 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb652e2-0c6a-4a08-8397-745b1c945660 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:04:42.390771 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb652e2-0c6a-4a08-8397-745b1c945660 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:04:42.408387 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1402/2813] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 01:04:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 01:04:42.410261 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d7566d-2c88-43e1-9a5d-0e6a08ca74c8 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:04:42.420006 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d7566d-2c88-43e1-9a5d-0e6a08ca74c8 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:04:42.425044 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0add46f-f8af-4a07-84f5-763d63e6a43e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:04:42.425660 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0add46f-f8af-4a07-84f5-763d63e6a43e admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:42.425933 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0add46f-f8af-4a07-84f5-763d63e6a43e admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:42.426321 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0add46f-f8af-4a07-84f5-763d63e6a43e admin admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:42.426989 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0add46f-f8af-4a07-84f5-763d63e6a43e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:04:42.429841 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0add46f-f8af-4a07-84f5-763d63e6a43e admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:04:42.472326 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1412/2814] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 01:04:42 2022] POST /identity/v3/projects => generated 373 bytes in 62 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 01:04:42.476757 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb652e2-0c6a-4a08-8397-745b1c945660 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:04:42.487192 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb652e2-0c6a-4a08-8397-745b1c945660 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:04:42.490723 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:04:42.546672 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48e68f9b-dda0-4b84-a31f-0f494c86f1bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:04:42.547328 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e68f9b-dda0-4b84-a31f-0f494c86f1bd admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:42.547609 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e68f9b-dda0-4b84-a31f-0f494c86f1bd admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:42.547875 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e68f9b-dda0-4b84-a31f-0f494c86f1bd admin admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:42.548495 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48e68f9b-dda0-4b84-a31f-0f494c86f1bd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:04:42.550402 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48e68f9b-dda0-4b84-a31f-0f494c86f1bd admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:04:42.571148 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1403/2815] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 01:04:42 2022] POST /identity/v3/users => generated 287 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 01:04:42.575915 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0add46f-f8af-4a07-84f5-763d63e6a43e admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:04:42.586005 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0add46f-f8af-4a07-84f5-763d63e6a43e admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:04:42.591293 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e91febd0-8311-4992-a46a-47ca3752919e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:04:42.591880 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e91febd0-8311-4992-a46a-47ca3752919e admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:42.592152 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e91febd0-8311-4992-a46a-47ca3752919e admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:42.592417 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e91febd0-8311-4992-a46a-47ca3752919e admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:42.593049 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e91febd0-8311-4992-a46a-47ca3752919e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:04:42.594623 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e91febd0-8311-4992-a46a-47ca3752919e admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:04:42.605227 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1413/2816] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 01:04:42 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 01:04:42.609334 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48e68f9b-dda0-4b84-a31f-0f494c86f1bd admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:04:42.619281 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48e68f9b-dda0-4b84-a31f-0f494c86f1bd admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:04:42.624049 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee34e6c2-61f9-4eab-895d-56b1b6311ac5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:04:42.624678 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee34e6c2-61f9-4eab-895d-56b1b6311ac5 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:42.624933 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee34e6c2-61f9-4eab-895d-56b1b6311ac5 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:42.625184 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee34e6c2-61f9-4eab-895d-56b1b6311ac5 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:42.625822 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee34e6c2-61f9-4eab-895d-56b1b6311ac5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:04:42.627456 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee34e6c2-61f9-4eab-895d-56b1b6311ac5 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:04:42.638513 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1404/2817] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 01:04:42 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 01:04:42.642560 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e91febd0-8311-4992-a46a-47ca3752919e admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:04:42.653202 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e91febd0-8311-4992-a46a-47ca3752919e admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:04:42.658118 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10b9907d-ade3-40b9-852b-26c562b49eec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:04:42.658758 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b9907d-ade3-40b9-852b-26c562b49eec admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:42.659073 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b9907d-ade3-40b9-852b-26c562b49eec admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:42.659346 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b9907d-ade3-40b9-852b-26c562b49eec admin admin] PATH_INFO: `/v3/projects/560fe0b4cf484ccdb4bb441eb14b632f/users/0b22fdbb52764cb1b0512a5e157720ae/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:42.673632 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10b9907d-ade3-40b9-852b-26c562b49eec admin admin] RBAC: Authorizing `identity:create_grant(project_id=560fe0b4cf484ccdb4bb441eb14b632f, user_id=0b22fdbb52764cb1b0512a5e157720ae, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:04:42.676313 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10b9907d-ade3-40b9-852b-26c562b49eec admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:04:42.689125 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1414/2818] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 01:04:42 2022] PUT /identity/v3/projects/560fe0b4cf484ccdb4bb441eb14b632f/users/0b22fdbb52764cb1b0512a5e157720ae/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 01:04:42.700855 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713b9494-fd7b-413e-8029-eed414a05365 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:42.701138 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713b9494-fd7b-413e-8029-eed414a05365 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:42.701412 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713b9494-fd7b-413e-8029-eed414a05365 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:42.730110 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-713b9494-fd7b-413e-8029-eed414a05365 None None] MFA Rules not processed for user `0b22fdbb52764cb1b0512a5e157720ae`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:04:42.731337 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-713b9494-fd7b-413e-8029-eed414a05365 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:04:42.916340 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1405/2819] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 01:04:42 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 217 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 01:04:42.927663 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10b9907d-ade3-40b9-852b-26c562b49eec admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:04:42.929357 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-10b9907d-ade3-40b9-852b-26c562b49eec admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:04:42.936353 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:04:43.004045 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10b9907d-ade3-40b9-852b-26c562b49eec admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:04:43.009484 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da81676-aca3-41df-a6da-dda9322a3863 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '835def0e6c6d4365a2767051509eb0d6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'ResellerAdmin', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:04:43.010087 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da81676-aca3-41df-a6da-dda9322a3863 service nova] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:43.010365 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da81676-aca3-41df-a6da-dda9322a3863 service nova] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:43.010636 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da81676-aca3-41df-a6da-dda9322a3863 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:43.018458 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0da81676-aca3-41df-a6da-dda9322a3863 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:04:43.020071 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0da81676-aca3-41df-a6da-dda9322a3863 service nova] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:04:43.029511 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1415/2820] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 01:04:42 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 01:04:45.703243 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713b9494-fd7b-413e-8029-eed414a05365 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:04:45.761568 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713b9494-fd7b-413e-8029-eed414a05365 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:04:45.766845 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df9067b3-e253-4997-b5f0-f1df7482a953 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:04:45.767458 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9067b3-e253-4997-b5f0-f1df7482a953 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:45.767729 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9067b3-e253-4997-b5f0-f1df7482a953 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:45.767991 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9067b3-e253-4997-b5f0-f1df7482a953 admin admin] PATH_INFO: `/v3/users/0b22fdbb52764cb1b0512a5e157720ae` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:45.769086 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df9067b3-e253-4997-b5f0-f1df7482a953 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0b22fdbb52764cb1b0512a5e157720ae)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:04:45.771005 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df9067b3-e253-4997-b5f0-f1df7482a953 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:04:45.786508 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=96,keystone:INSERT=3,keystone:DELETE=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:04:45.819291 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-df9067b3-e253-4997-b5f0-f1df7482a953 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0b22fdbb52764cb1b0512a5e157720ae'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:04:45.819935 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-df9067b3-e253-4997-b5f0-f1df7482a953 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0b22fdbb52764cb1b0512a5e157720ae'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:04:45.820555 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-df9067b3-e253-4997-b5f0-f1df7482a953 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0b22fdbb52764cb1b0512a5e157720ae'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:04:45.828239 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-df9067b3-e253-4997-b5f0-f1df7482a953 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0b22fdbb52764cb1b0512a5e157720ae'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:04:45.839902 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-df9067b3-e253-4997-b5f0-f1df7482a953 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0b22fdbb52764cb1b0512a5e157720ae'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:04:45.853008 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1406/2821] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 01:04:45 2022] DELETE /identity/v3/users/0b22fdbb52764cb1b0512a5e157720ae => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 01:04:45.857755 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e175389a-7743-48e9-8a9a-e12457018b81 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:45.858254 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e175389a-7743-48e9-8a9a-e12457018b81 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:45.858371 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e175389a-7743-48e9-8a9a-e12457018b81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:45.877323 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-e175389a-7743-48e9-8a9a-e12457018b81 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:04:45.878507 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-e175389a-7743-48e9-8a9a-e12457018b81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:04:45.936208 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1416/2822] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 01:04:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 01:04:45.945882 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df9067b3-e253-4997-b5f0-f1df7482a953 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:04:45.947315 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-df9067b3-e253-4997-b5f0-f1df7482a953 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:04:46.005026 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df9067b3-e253-4997-b5f0-f1df7482a953 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:04:46.008731 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6821b73c-2372-4826-a2fa-77d7fee27764 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:04:46.009339 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6821b73c-2372-4826-a2fa-77d7fee27764 service neutron] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:46.009618 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6821b73c-2372-4826-a2fa-77d7fee27764 service neutron] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:46.009890 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6821b73c-2372-4826-a2fa-77d7fee27764 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:46.019078 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6821b73c-2372-4826-a2fa-77d7fee27764 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:04:46.020465 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6821b73c-2372-4826-a2fa-77d7fee27764 service neutron] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:04:46.030254 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1407/2823] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 01:04:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 01:04:46.297997 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e175389a-7743-48e9-8a9a-e12457018b81 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:04:46.299465 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-e175389a-7743-48e9-8a9a-e12457018b81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:04:46.312946 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e175389a-7743-48e9-8a9a-e12457018b81 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:04:46.317642 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7547cdf9-f7ae-47da-9153-5711710edc78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:04:46.318281 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7547cdf9-f7ae-47da-9153-5711710edc78 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:46.318559 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7547cdf9-f7ae-47da-9153-5711710edc78 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:46.318862 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7547cdf9-f7ae-47da-9153-5711710edc78 admin admin] PATH_INFO: `/v3/projects/560fe0b4cf484ccdb4bb441eb14b632f` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:46.319997 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7547cdf9-f7ae-47da-9153-5711710edc78 admin admin] RBAC: Authorizing `identity:delete_project(project_id=560fe0b4cf484ccdb4bb441eb14b632f)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:04:46.321916 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7547cdf9-f7ae-47da-9153-5711710edc78 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:04:46.371611 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-7547cdf9-f7ae-47da-9153-5711710edc78 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '560fe0b4cf484ccdb4bb441eb14b632f', 'request_id': 'req-7547cdf9-f7ae-47da-9153-5711710edc78'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:04:46.377836 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-7547cdf9-f7ae-47da-9153-5711710edc78 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '560fe0b4cf484ccdb4bb441eb14b632f', 'request_id': 'req-7547cdf9-f7ae-47da-9153-5711710edc78'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:04:46.385214 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-7547cdf9-f7ae-47da-9153-5711710edc78 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '560fe0b4cf484ccdb4bb441eb14b632f', 'request_id': 'req-7547cdf9-f7ae-47da-9153-5711710edc78'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:04:46.396211 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-7547cdf9-f7ae-47da-9153-5711710edc78 admin admin] The token cache is being invalidate because project 560fe0b4cf484ccdb4bb441eb14b632f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87924) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 01:04:46.396499 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-7547cdf9-f7ae-47da-9153-5711710edc78 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:04:46.397139 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-7547cdf9-f7ae-47da-9153-5711710edc78 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:04:46.398617 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1417/2824] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 01:04:46 2022] DELETE /identity/v3/projects/560fe0b4cf484ccdb4bb441eb14b632f => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 01:04:46.404017 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868c979b-573d-4e65-908d-1b9f8aa22855 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:46.404305 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868c979b-573d-4e65-908d-1b9f8aa22855 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:46.404578 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868c979b-573d-4e65-908d-1b9f8aa22855 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:46.423998 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-868c979b-573d-4e65-908d-1b9f8aa22855 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:04:46.425196 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-868c979b-573d-4e65-908d-1b9f8aa22855 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:04:46.483855 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1408/2825] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 01:04:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 01:04:46.491646 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08479df4-8403-42b3-8470-1bba99861150 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:46.491932 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08479df4-8403-42b3-8470-1bba99861150 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:46.492207 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08479df4-8403-42b3-8470-1bba99861150 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:46.512879 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-08479df4-8403-42b3-8470-1bba99861150 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:04:46.514090 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-08479df4-8403-42b3-8470-1bba99861150 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:04:46.522184 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1418/2826] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 01:04:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 01:04:46.526279 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-868c979b-573d-4e65-908d-1b9f8aa22855 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:04:46.536370 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-868c979b-573d-4e65-908d-1b9f8aa22855 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:04:46.540889 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd260789-a2f0-4f97-ba28-335fc22ab403 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:04:46.541408 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd260789-a2f0-4f97-ba28-335fc22ab403 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:46.541677 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd260789-a2f0-4f97-ba28-335fc22ab403 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:46.541938 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd260789-a2f0-4f97-ba28-335fc22ab403 admin admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:46.542567 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd260789-a2f0-4f97-ba28-335fc22ab403 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 01:04:46.542900 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd260789-a2f0-4f97-ba28-335fc22ab403 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:04:46.544829 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd260789-a2f0-4f97-ba28-335fc22ab403 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:04:46.556757 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1409/2827] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 01:04:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 01:04:46.561670 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08479df4-8403-42b3-8470-1bba99861150 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:04:46.570892 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08479df4-8403-42b3-8470-1bba99861150 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:04:46.576495 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e0780f4-9465-4639-812d-69a0fc4a5583 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:04:46.577084 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e0780f4-9465-4639-812d-69a0fc4a5583 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:46.577360 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e0780f4-9465-4639-812d-69a0fc4a5583 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:46.577634 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e0780f4-9465-4639-812d-69a0fc4a5583 admin admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:46.578276 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e0780f4-9465-4639-812d-69a0fc4a5583 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:04:46.580205 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e0780f4-9465-4639-812d-69a0fc4a5583 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:04:46.584668 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:04:46.608081 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1419/2828] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 01:04:46 2022] POST /identity/v3/projects => generated 377 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 01:04:46.612234 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd260789-a2f0-4f97-ba28-335fc22ab403 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:04:46.622657 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd260789-a2f0-4f97-ba28-335fc22ab403 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:04:46.643215 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:DELETE=9,keystone:SELECT=90,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:04:46.679816 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0090cfa6-74ba-42bd-af23-6a4b3d544167 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:04:46.680433 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0090cfa6-74ba-42bd-af23-6a4b3d544167 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:46.680704 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0090cfa6-74ba-42bd-af23-6a4b3d544167 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:46.680970 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0090cfa6-74ba-42bd-af23-6a4b3d544167 admin admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:46.681591 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0090cfa6-74ba-42bd-af23-6a4b3d544167 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:04:46.683530 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0090cfa6-74ba-42bd-af23-6a4b3d544167 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:04:46.706404 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1410/2829] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 01:04:46 2022] POST /identity/v3/users => generated 289 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 01:04:46.710513 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e0780f4-9465-4639-812d-69a0fc4a5583 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:04:46.722003 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e0780f4-9465-4639-812d-69a0fc4a5583 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:04:46.726592 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8544100c-83e1-4793-977b-545ac860cdff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:04:46.727267 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8544100c-83e1-4793-977b-545ac860cdff admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:46.727507 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8544100c-83e1-4793-977b-545ac860cdff admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:46.727745 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8544100c-83e1-4793-977b-545ac860cdff admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:46.728425 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8544100c-83e1-4793-977b-545ac860cdff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:04:46.729991 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8544100c-83e1-4793-977b-545ac860cdff admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:04:46.740720 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1420/2830] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 01:04:46 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 01:04:46.744632 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0090cfa6-74ba-42bd-af23-6a4b3d544167 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:04:46.755221 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0090cfa6-74ba-42bd-af23-6a4b3d544167 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:04:46.759767 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b19135a7-50e0-400f-8908-d709e51eaacb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:04:46.760344 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19135a7-50e0-400f-8908-d709e51eaacb admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:46.760614 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19135a7-50e0-400f-8908-d709e51eaacb admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:46.760878 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19135a7-50e0-400f-8908-d709e51eaacb admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:46.761500 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b19135a7-50e0-400f-8908-d709e51eaacb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:04:46.763952 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b19135a7-50e0-400f-8908-d709e51eaacb admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:04:46.773526 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1411/2831] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 01:04:46 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 01:04:46.777627 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8544100c-83e1-4793-977b-545ac860cdff admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:04:46.788078 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8544100c-83e1-4793-977b-545ac860cdff admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:04:46.792709 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08215b0c-f646-4c04-9315-f2bddd867a00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:04:46.793850 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08215b0c-f646-4c04-9315-f2bddd867a00 admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:46.794348 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08215b0c-f646-4c04-9315-f2bddd867a00 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:46.794659 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08215b0c-f646-4c04-9315-f2bddd867a00 admin admin] PATH_INFO: `/v3/projects/f1e0123689e7474e9847599b68bf4103/users/7bf4e992796e48448a3af1fa9e402eab/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:46.809680 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08215b0c-f646-4c04-9315-f2bddd867a00 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f1e0123689e7474e9847599b68bf4103, user_id=7bf4e992796e48448a3af1fa9e402eab, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:04:46.812416 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08215b0c-f646-4c04-9315-f2bddd867a00 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:04:46.824919 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1421/2832] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 01:04:46 2022] PUT /identity/v3/projects/f1e0123689e7474e9847599b68bf4103/users/7bf4e992796e48448a3af1fa9e402eab/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 01:04:46.836913 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d239e5d-a206-4af7-b4c7-53b45df78787 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:46.837212 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d239e5d-a206-4af7-b4c7-53b45df78787 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:46.837479 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d239e5d-a206-4af7-b4c7-53b45df78787 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:46.865237 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-9d239e5d-a206-4af7-b4c7-53b45df78787 None None] MFA Rules not processed for user `7bf4e992796e48448a3af1fa9e402eab`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:04:46.866432 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-9d239e5d-a206-4af7-b4c7-53b45df78787 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:04:46.928997 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1412/2833] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 01:04:46 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 01:04:46.933590 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08215b0c-f646-4c04-9315-f2bddd867a00 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:04:46.995889 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08215b0c-f646-4c04-9315-f2bddd867a00 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:04:47.000570 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51a27404-5125-4666-93a1-a4b985c71b7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:04:47.001165 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a27404-5125-4666-93a1-a4b985c71b7d admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:47.001443 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a27404-5125-4666-93a1-a4b985c71b7d admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:47.001712 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a27404-5125-4666-93a1-a4b985c71b7d admin admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:47.002346 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51a27404-5125-4666-93a1-a4b985c71b7d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:04:47.004265 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51a27404-5125-4666-93a1-a4b985c71b7d admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:04:47.031408 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1422/2834] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 01:04:46 2022] POST /identity/v3/projects => generated 381 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 01:04:47.035697 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d239e5d-a206-4af7-b4c7-53b45df78787 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:04:47.046679 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d239e5d-a206-4af7-b4c7-53b45df78787 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:04:47.102949 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9643df4-79be-451c-99a4-e73714521a83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:04:47.103559 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9643df4-79be-451c-99a4-e73714521a83 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:47.103838 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9643df4-79be-451c-99a4-e73714521a83 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:47.104108 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9643df4-79be-451c-99a4-e73714521a83 admin admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:47.104750 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9643df4-79be-451c-99a4-e73714521a83 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:04:47.106998 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9643df4-79be-451c-99a4-e73714521a83 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:04:47.127819 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1413/2835] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 01:04:47 2022] POST /identity/v3/users => generated 297 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 01:04:47.132017 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51a27404-5125-4666-93a1-a4b985c71b7d admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:04:47.142653 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51a27404-5125-4666-93a1-a4b985c71b7d admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:04:47.147286 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6879285e-f5c4-49f1-b019-101935b48f51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:04:47.147877 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6879285e-f5c4-49f1-b019-101935b48f51 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:47.148164 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6879285e-f5c4-49f1-b019-101935b48f51 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:47.148434 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6879285e-f5c4-49f1-b019-101935b48f51 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:47.149074 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6879285e-f5c4-49f1-b019-101935b48f51 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:04:47.150660 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6879285e-f5c4-49f1-b019-101935b48f51 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:04:47.162259 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1423/2836] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 01:04:47 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 11 01:04:47.166222 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9643df4-79be-451c-99a4-e73714521a83 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:04:47.176476 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9643df4-79be-451c-99a4-e73714521a83 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:04:47.181098 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d530601-a112-4155-9d85-439910fa630f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:04:47.181861 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d530601-a112-4155-9d85-439910fa630f admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:47.182139 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d530601-a112-4155-9d85-439910fa630f admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:47.182410 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d530601-a112-4155-9d85-439910fa630f admin admin] PATH_INFO: `/v3/projects/efceec0d9e3f4c409faa60f4926573a4/users/31a2f61f772e4805a74ea46a6a5df5b8/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:47.198813 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d530601-a112-4155-9d85-439910fa630f admin admin] RBAC: Authorizing `identity:create_grant(project_id=efceec0d9e3f4c409faa60f4926573a4, user_id=31a2f61f772e4805a74ea46a6a5df5b8, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:04:47.202054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d530601-a112-4155-9d85-439910fa630f admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:04:47.209800 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:04:47.216157 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1414/2837] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 01:04:47 2022] PUT /identity/v3/projects/efceec0d9e3f4c409faa60f4926573a4/users/31a2f61f772e4805a74ea46a6a5df5b8/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 01:04:47.220091 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6879285e-f5c4-49f1-b019-101935b48f51 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:04:47.230059 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6879285e-f5c4-49f1-b019-101935b48f51 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:04:47.285670 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b810fc-cf47-4d03-b2dd-d16857f92def admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:04:47.286269 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b810fc-cf47-4d03-b2dd-d16857f92def admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:47.286549 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b810fc-cf47-4d03-b2dd-d16857f92def admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:47.286849 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b810fc-cf47-4d03-b2dd-d16857f92def admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:47.287491 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66b810fc-cf47-4d03-b2dd-d16857f92def admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:04:47.289107 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66b810fc-cf47-4d03-b2dd-d16857f92def admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:04:47.298913 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1424/2838] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 01:04:47 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 01:04:47.303002 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d530601-a112-4155-9d85-439910fa630f admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:04:47.313968 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d530601-a112-4155-9d85-439910fa630f admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:04:47.318480 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c911a26-ea8e-454d-ae17-1881061f8d18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:04:47.319150 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c911a26-ea8e-454d-ae17-1881061f8d18 admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:47.319446 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c911a26-ea8e-454d-ae17-1881061f8d18 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:47.319709 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c911a26-ea8e-454d-ae17-1881061f8d18 admin admin] PATH_INFO: `/v3/projects/efceec0d9e3f4c409faa60f4926573a4/users/31a2f61f772e4805a74ea46a6a5df5b8/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:47.321584 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c911a26-ea8e-454d-ae17-1881061f8d18 admin admin] RBAC: Authorizing `identity:create_grant(project_id=efceec0d9e3f4c409faa60f4926573a4, user_id=31a2f61f772e4805a74ea46a6a5df5b8, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:04:47.324207 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c911a26-ea8e-454d-ae17-1881061f8d18 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:04:47.335090 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1415/2839] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 01:04:47 2022] PUT /identity/v3/projects/efceec0d9e3f4c409faa60f4926573a4/users/31a2f61f772e4805a74ea46a6a5df5b8/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 01:04:47.345464 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9438d3-0512-4b93-9ae3-3408119e6442 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:47.345747 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9438d3-0512-4b93-9ae3-3408119e6442 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:47.346020 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9438d3-0512-4b93-9ae3-3408119e6442 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:47.374977 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-3a9438d3-0512-4b93-9ae3-3408119e6442 None None] MFA Rules not processed for user `31a2f61f772e4805a74ea46a6a5df5b8`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:04:47.376165 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-3a9438d3-0512-4b93-9ae3-3408119e6442 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:04:47.384108 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:04:47.445474 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1425/2840] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 01:04:47 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 01:04:47.458197 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c911a26-ea8e-454d-ae17-1881061f8d18 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:04:47.459980 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-7c911a26-ea8e-454d-ae17-1881061f8d18 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:04:47.516617 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c911a26-ea8e-454d-ae17-1881061f8d18 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:04:47.520309 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8bfd73a-5fa1-4473-b7b6-ca4f5937cf10 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:04:47.520897 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bfd73a-5fa1-4473-b7b6-ca4f5937cf10 service cinder] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:47.521178 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bfd73a-5fa1-4473-b7b6-ca4f5937cf10 service cinder] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:47.521446 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bfd73a-5fa1-4473-b7b6-ca4f5937cf10 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:47.530365 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8bfd73a-5fa1-4473-b7b6-ca4f5937cf10 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:04:47.531830 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8bfd73a-5fa1-4473-b7b6-ca4f5937cf10 service cinder] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:04:47.541362 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1416/2841] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 01:04:47 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 01:04:47.615196 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a9438d3-0512-4b93-9ae3-3408119e6442 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:04:47.625991 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a9438d3-0512-4b93-9ae3-3408119e6442 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:04:47.629606 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1db3408-9e45-47ad-9dfb-b5300e25aa00 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:04:47.630232 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1db3408-9e45-47ad-9dfb-b5300e25aa00 service cinder] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:47.630506 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1db3408-9e45-47ad-9dfb-b5300e25aa00 service cinder] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:47.630820 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1db3408-9e45-47ad-9dfb-b5300e25aa00 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:47.686452 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1db3408-9e45-47ad-9dfb-b5300e25aa00 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:04:47.687926 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1db3408-9e45-47ad-9dfb-b5300e25aa00 service cinder] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:04:47.696703 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1426/2842] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 01:04:47 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 01:04:56.541272 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8bfd73a-5fa1-4473-b7b6-ca4f5937cf10 service cinder] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:04:56.602501 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8bfd73a-5fa1-4473-b7b6-ca4f5937cf10 service cinder] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:04:56.607566 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e9b26d3-23af-4681-a713-7e14e1f360b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:04:56.608132 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e9b26d3-23af-4681-a713-7e14e1f360b7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:56.608408 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e9b26d3-23af-4681-a713-7e14e1f360b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:56.608676 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e9b26d3-23af-4681-a713-7e14e1f360b7 admin admin] PATH_INFO: `/v3/users/7bf4e992796e48448a3af1fa9e402eab` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:56.609783 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e9b26d3-23af-4681-a713-7e14e1f360b7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7bf4e992796e48448a3af1fa9e402eab)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:04:56.611718 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e9b26d3-23af-4681-a713-7e14e1f360b7 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:04:56.656251 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-0e9b26d3-23af-4681-a713-7e14e1f360b7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7bf4e992796e48448a3af1fa9e402eab'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:04:56.656882 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-0e9b26d3-23af-4681-a713-7e14e1f360b7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7bf4e992796e48448a3af1fa9e402eab'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:04:56.657483 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-0e9b26d3-23af-4681-a713-7e14e1f360b7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7bf4e992796e48448a3af1fa9e402eab'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:04:56.664584 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-0e9b26d3-23af-4681-a713-7e14e1f360b7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7bf4e992796e48448a3af1fa9e402eab'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:04:56.677012 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-0e9b26d3-23af-4681-a713-7e14e1f360b7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7bf4e992796e48448a3af1fa9e402eab'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:04:56.690671 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1417/2843] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 01:04:56 2022] DELETE /identity/v3/users/7bf4e992796e48448a3af1fa9e402eab => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 01:04:56.695218 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-433232ed-97dd-45b3-ac8d-4f1741932537 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:56.695717 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-433232ed-97dd-45b3-ac8d-4f1741932537 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:56.695831 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-433232ed-97dd-45b3-ac8d-4f1741932537 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:56.714472 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-433232ed-97dd-45b3-ac8d-4f1741932537 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:04:56.715696 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-433232ed-97dd-45b3-ac8d-4f1741932537 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:04:56.773926 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1427/2844] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 01:04:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 01:04:56.783960 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e9b26d3-23af-4681-a713-7e14e1f360b7 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:04:56.785333 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-0e9b26d3-23af-4681-a713-7e14e1f360b7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:04:56.795902 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:04:56.845017 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e9b26d3-23af-4681-a713-7e14e1f360b7 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:04:56.848689 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-804e9109-7ba4-4fce-9d8a-52c26fbc07a8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:04:56.849290 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804e9109-7ba4-4fce-9d8a-52c26fbc07a8 service neutron] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:56.849570 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804e9109-7ba4-4fce-9d8a-52c26fbc07a8 service neutron] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:56.849843 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804e9109-7ba4-4fce-9d8a-52c26fbc07a8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:56.859513 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-804e9109-7ba4-4fce-9d8a-52c26fbc07a8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:04:56.860900 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-804e9109-7ba4-4fce-9d8a-52c26fbc07a8 service neutron] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:04:56.870316 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1418/2845] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 01:04:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 01:04:57.134408 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-433232ed-97dd-45b3-ac8d-4f1741932537 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:04:57.135973 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-433232ed-97dd-45b3-ac8d-4f1741932537 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:04:57.148651 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-433232ed-97dd-45b3-ac8d-4f1741932537 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:04:57.153417 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cec98e8-e6c9-4b4e-aa89-5936c5c864f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:04:57.154756 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cec98e8-e6c9-4b4e-aa89-5936c5c864f0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:57.155127 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cec98e8-e6c9-4b4e-aa89-5936c5c864f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:57.155460 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cec98e8-e6c9-4b4e-aa89-5936c5c864f0 admin admin] PATH_INFO: `/v3/projects/f1e0123689e7474e9847599b68bf4103` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:57.156614 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cec98e8-e6c9-4b4e-aa89-5936c5c864f0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f1e0123689e7474e9847599b68bf4103)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:04:57.158913 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cec98e8-e6c9-4b4e-aa89-5936c5c864f0 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:04:57.202669 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:04:57.210428 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-7cec98e8-e6c9-4b4e-aa89-5936c5c864f0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f1e0123689e7474e9847599b68bf4103', 'request_id': 'req-7cec98e8-e6c9-4b4e-aa89-5936c5c864f0'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:04:57.217174 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-7cec98e8-e6c9-4b4e-aa89-5936c5c864f0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f1e0123689e7474e9847599b68bf4103', 'request_id': 'req-7cec98e8-e6c9-4b4e-aa89-5936c5c864f0'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:04:57.224526 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-7cec98e8-e6c9-4b4e-aa89-5936c5c864f0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f1e0123689e7474e9847599b68bf4103', 'request_id': 'req-7cec98e8-e6c9-4b4e-aa89-5936c5c864f0'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:04:57.235853 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-7cec98e8-e6c9-4b4e-aa89-5936c5c864f0 admin admin] The token cache is being invalidate because project f1e0123689e7474e9847599b68bf4103 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87924) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 01:04:57.236160 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-7cec98e8-e6c9-4b4e-aa89-5936c5c864f0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:04:57.236801 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-7cec98e8-e6c9-4b4e-aa89-5936c5c864f0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:04:57.238266 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1428/2846] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 01:04:57 2022] DELETE /identity/v3/projects/f1e0123689e7474e9847599b68bf4103 => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 01:04:57.242312 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-804e9109-7ba4-4fce-9d8a-52c26fbc07a8 service neutron] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:04:57.252723 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-804e9109-7ba4-4fce-9d8a-52c26fbc07a8 service neutron] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:04:57.324513 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aba7f75-df0f-46cf-8646-a5140e3eb86f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:04:57.326171 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aba7f75-df0f-46cf-8646-a5140e3eb86f admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:57.326447 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aba7f75-df0f-46cf-8646-a5140e3eb86f admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:57.326713 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aba7f75-df0f-46cf-8646-a5140e3eb86f admin admin] PATH_INFO: `/v3/users/31a2f61f772e4805a74ea46a6a5df5b8` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:57.327888 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aba7f75-df0f-46cf-8646-a5140e3eb86f admin admin] RBAC: Authorizing `identity:delete_user(user_id=31a2f61f772e4805a74ea46a6a5df5b8)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:04:57.329984 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aba7f75-df0f-46cf-8646-a5140e3eb86f admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:04:57.386512 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-9aba7f75-df0f-46cf-8646-a5140e3eb86f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '31a2f61f772e4805a74ea46a6a5df5b8'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:04:57.387116 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-9aba7f75-df0f-46cf-8646-a5140e3eb86f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '31a2f61f772e4805a74ea46a6a5df5b8'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:04:57.387716 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-9aba7f75-df0f-46cf-8646-a5140e3eb86f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '31a2f61f772e4805a74ea46a6a5df5b8'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:04:57.396354 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-9aba7f75-df0f-46cf-8646-a5140e3eb86f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '31a2f61f772e4805a74ea46a6a5df5b8'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:04:57.409470 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-9aba7f75-df0f-46cf-8646-a5140e3eb86f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '31a2f61f772e4805a74ea46a6a5df5b8'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:04:57.426338 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1419/2847] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 01:04:57 2022] DELETE /identity/v3/users/31a2f61f772e4805a74ea46a6a5df5b8 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 01:04:57.740368 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cec98e8-e6c9-4b4e-aa89-5936c5c864f0 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:04:57.743850 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-7cec98e8-e6c9-4b4e-aa89-5936c5c864f0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:04:57.758572 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cec98e8-e6c9-4b4e-aa89-5936c5c864f0 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:04:57.827655 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d55247cb-0f9a-4249-86ac-311ba214704c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:04:57.828287 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55247cb-0f9a-4249-86ac-311ba214704c admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:57.828562 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55247cb-0f9a-4249-86ac-311ba214704c admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:57.828831 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55247cb-0f9a-4249-86ac-311ba214704c admin admin] PATH_INFO: `/v3/projects/efceec0d9e3f4c409faa60f4926573a4` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:57.829959 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d55247cb-0f9a-4249-86ac-311ba214704c admin admin] RBAC: Authorizing `identity:delete_project(project_id=efceec0d9e3f4c409faa60f4926573a4)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:04:57.831895 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d55247cb-0f9a-4249-86ac-311ba214704c admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:04:57.880209 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-d55247cb-0f9a-4249-86ac-311ba214704c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'efceec0d9e3f4c409faa60f4926573a4', 'request_id': 'req-d55247cb-0f9a-4249-86ac-311ba214704c'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:04:57.887802 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-d55247cb-0f9a-4249-86ac-311ba214704c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'efceec0d9e3f4c409faa60f4926573a4', 'request_id': 'req-d55247cb-0f9a-4249-86ac-311ba214704c'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:04:57.899088 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-d55247cb-0f9a-4249-86ac-311ba214704c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'efceec0d9e3f4c409faa60f4926573a4', 'request_id': 'req-d55247cb-0f9a-4249-86ac-311ba214704c'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:04:57.913617 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-d55247cb-0f9a-4249-86ac-311ba214704c admin admin] The token cache is being invalidate because project efceec0d9e3f4c409faa60f4926573a4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87924) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 01:04:57.915540 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-d55247cb-0f9a-4249-86ac-311ba214704c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:04:57.916218 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-d55247cb-0f9a-4249-86ac-311ba214704c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:04:57.917775 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1429/2848] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 01:04:57 2022] DELETE /identity/v3/projects/efceec0d9e3f4c409faa60f4926573a4 => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 01:04:57.925629 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a2b7f8-35bd-4548-bfa3-67f89be4d12c None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:57.926158 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a2b7f8-35bd-4548-bfa3-67f89be4d12c None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:57.926158 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a2b7f8-35bd-4548-bfa3-67f89be4d12c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:57.946827 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-a7a2b7f8-35bd-4548-bfa3-67f89be4d12c None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:04:57.948037 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-a7a2b7f8-35bd-4548-bfa3-67f89be4d12c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:04:57.973692 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:04:58.016573 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1420/2849] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 01:04:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 01:04:58.026365 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb523f66-859e-474b-99d6-363123d724f2 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:58.026646 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb523f66-859e-474b-99d6-363123d724f2 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:58.026952 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb523f66-859e-474b-99d6-363123d724f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:58.047259 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-cb523f66-859e-474b-99d6-363123d724f2 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:04:58.048454 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-cb523f66-859e-474b-99d6-363123d724f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:04:58.056225 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1430/2850] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 01:04:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 01:04:58.061070 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a2b7f8-35bd-4548-bfa3-67f89be4d12c None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:04:58.070711 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a2b7f8-35bd-4548-bfa3-67f89be4d12c None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:04:58.075274 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-097f1bb0-0515-4e43-b2f3-98eea0b11149 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:04:58.075893 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097f1bb0-0515-4e43-b2f3-98eea0b11149 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:58.076350 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097f1bb0-0515-4e43-b2f3-98eea0b11149 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:58.076628 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097f1bb0-0515-4e43-b2f3-98eea0b11149 admin admin] PATH_INFO: `/v3/domains` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:58.077275 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-097f1bb0-0515-4e43-b2f3-98eea0b11149 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87923) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 01:04:58.077576 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-097f1bb0-0515-4e43-b2f3-98eea0b11149 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:04:58.079201 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-097f1bb0-0515-4e43-b2f3-98eea0b11149 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:04:58.090862 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1421/2851] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 01:04:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 01:04:58.095370 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb523f66-859e-474b-99d6-363123d724f2 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:04:58.105294 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb523f66-859e-474b-99d6-363123d724f2 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:04:58.110759 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d5edc0-c7b5-4ccf-bf35-f952fb70790d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:04:58.111377 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d5edc0-c7b5-4ccf-bf35-f952fb70790d admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:58.111649 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d5edc0-c7b5-4ccf-bf35-f952fb70790d admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:58.111914 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d5edc0-c7b5-4ccf-bf35-f952fb70790d admin admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:58.112533 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26d5edc0-c7b5-4ccf-bf35-f952fb70790d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:04:58.114404 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26d5edc0-c7b5-4ccf-bf35-f952fb70790d admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:04:58.142353 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1431/2852] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 01:04:58 2022] POST /identity/v3/projects => generated 381 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 01:04:58.146584 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-097f1bb0-0515-4e43-b2f3-98eea0b11149 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:04:58.156638 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-097f1bb0-0515-4e43-b2f3-98eea0b11149 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:04:58.212163 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a97099-17ef-4b4f-91c5-032bf559acd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:04:58.212832 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a97099-17ef-4b4f-91c5-032bf559acd6 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:58.213107 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a97099-17ef-4b4f-91c5-032bf559acd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:58.213375 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a97099-17ef-4b4f-91c5-032bf559acd6 admin admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:58.213997 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43a97099-17ef-4b4f-91c5-032bf559acd6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:04:58.215932 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43a97099-17ef-4b4f-91c5-032bf559acd6 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:04:58.237963 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1422/2853] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 01:04:58 2022] POST /identity/v3/users => generated 291 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 01:04:58.242122 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d5edc0-c7b5-4ccf-bf35-f952fb70790d admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:04:58.253125 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d5edc0-c7b5-4ccf-bf35-f952fb70790d admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:04:58.257891 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c36976-a66a-4190-ab8e-2665d3116dd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:04:58.258483 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c36976-a66a-4190-ab8e-2665d3116dd8 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:58.258759 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c36976-a66a-4190-ab8e-2665d3116dd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:58.259053 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c36976-a66a-4190-ab8e-2665d3116dd8 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:58.259742 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4c36976-a66a-4190-ab8e-2665d3116dd8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:04:58.261357 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4c36976-a66a-4190-ab8e-2665d3116dd8 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:04:58.273483 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1432/2854] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 01:04:58 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 01:04:58.277379 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a97099-17ef-4b4f-91c5-032bf559acd6 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:04:58.287794 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a97099-17ef-4b4f-91c5-032bf559acd6 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:04:58.292490 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c811f565-40e4-45f3-9902-1864b81abb56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:04:58.293075 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c811f565-40e4-45f3-9902-1864b81abb56 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:58.293353 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c811f565-40e4-45f3-9902-1864b81abb56 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:58.293623 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c811f565-40e4-45f3-9902-1864b81abb56 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:58.294747 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c811f565-40e4-45f3-9902-1864b81abb56 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:04:58.296361 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c811f565-40e4-45f3-9902-1864b81abb56 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:04:58.306684 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1423/2855] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 01:04:58 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 01:04:58.311595 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c36976-a66a-4190-ab8e-2665d3116dd8 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:04:58.321660 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c36976-a66a-4190-ab8e-2665d3116dd8 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:04:58.326887 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d93d16-0c10-4eeb-886e-482829f25c4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:04:58.327541 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d93d16-0c10-4eeb-886e-482829f25c4d admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:58.327823 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d93d16-0c10-4eeb-886e-482829f25c4d admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:58.328101 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d93d16-0c10-4eeb-886e-482829f25c4d admin admin] PATH_INFO: `/v3/projects/a0a17562cbbf41909c74b6319ac0898b/users/cebe9e04cb2848f39e99e7195fcf5dc1/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:58.343040 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0d93d16-0c10-4eeb-886e-482829f25c4d admin admin] RBAC: Authorizing `identity:create_grant(project_id=a0a17562cbbf41909c74b6319ac0898b, user_id=cebe9e04cb2848f39e99e7195fcf5dc1, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:04:58.345710 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0d93d16-0c10-4eeb-886e-482829f25c4d admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:04:58.347915 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=86,keystone:DELETE=11,keystone:INSERT=3 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:04:58.361210 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1433/2856] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 01:04:58 2022] PUT /identity/v3/projects/a0a17562cbbf41909c74b6319ac0898b/users/cebe9e04cb2848f39e99e7195fcf5dc1/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 01:04:58.373009 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8171180a-95b4-4111-8ac0-b76e5c7d35d4 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:58.373470 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8171180a-95b4-4111-8ac0-b76e5c7d35d4 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:58.373898 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8171180a-95b4-4111-8ac0-b76e5c7d35d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:58.408714 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-8171180a-95b4-4111-8ac0-b76e5c7d35d4 None None] MFA Rules not processed for user `cebe9e04cb2848f39e99e7195fcf5dc1`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:04:58.409907 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-8171180a-95b4-4111-8ac0-b76e5c7d35d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:04:58.473529 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1424/2857] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 01:04:58 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 01:04:58.478080 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d93d16-0c10-4eeb-886e-482829f25c4d admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:04:58.538750 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d93d16-0c10-4eeb-886e-482829f25c4d admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:04:58.543535 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f41f9e-6195-424d-b05a-40f2c7635f18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:04:58.544135 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f41f9e-6195-424d-b05a-40f2c7635f18 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:58.545080 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f41f9e-6195-424d-b05a-40f2c7635f18 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:58.545357 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f41f9e-6195-424d-b05a-40f2c7635f18 admin admin] PATH_INFO: `/v3/projects` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:58.546007 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26f41f9e-6195-424d-b05a-40f2c7635f18 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:04:58.547951 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26f41f9e-6195-424d-b05a-40f2c7635f18 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:04:58.573135 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1434/2858] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 01:04:58 2022] POST /identity/v3/projects => generated 377 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 11 01:04:58.577305 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8171180a-95b4-4111-8ac0-b76e5c7d35d4 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:04:58.587899 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8171180a-95b4-4111-8ac0-b76e5c7d35d4 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:04:58.608183 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:04:58.644621 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6afe3d0c-b55e-4391-8904-101df8ee58cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:04:58.645227 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6afe3d0c-b55e-4391-8904-101df8ee58cc admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:58.645505 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6afe3d0c-b55e-4391-8904-101df8ee58cc admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:58.645775 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6afe3d0c-b55e-4391-8904-101df8ee58cc admin admin] PATH_INFO: `/v3/users` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:58.646411 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6afe3d0c-b55e-4391-8904-101df8ee58cc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:04:58.648353 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6afe3d0c-b55e-4391-8904-101df8ee58cc admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:04:58.670734 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1425/2859] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 01:04:58 2022] POST /identity/v3/users => generated 295 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 01:04:58.674765 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f41f9e-6195-424d-b05a-40f2c7635f18 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:04:58.684775 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f41f9e-6195-424d-b05a-40f2c7635f18 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:04:58.689419 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567d2ce5-d598-43aa-ae80-c2af7a810d22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:04:58.690008 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567d2ce5-d598-43aa-ae80-c2af7a810d22 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:58.690284 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567d2ce5-d598-43aa-ae80-c2af7a810d22 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:58.690612 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567d2ce5-d598-43aa-ae80-c2af7a810d22 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:58.691280 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-567d2ce5-d598-43aa-ae80-c2af7a810d22 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:04:58.692851 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-567d2ce5-d598-43aa-ae80-c2af7a810d22 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:04:58.703236 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1435/2860] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 01:04:58 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 01:04:58.707224 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6afe3d0c-b55e-4391-8904-101df8ee58cc admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:04:58.717500 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6afe3d0c-b55e-4391-8904-101df8ee58cc admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:04:58.721985 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dc48e30-1474-4215-8c20-2c0bd454df3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:04:58.722618 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc48e30-1474-4215-8c20-2c0bd454df3e admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:58.722926 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc48e30-1474-4215-8c20-2c0bd454df3e admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:58.723194 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc48e30-1474-4215-8c20-2c0bd454df3e admin admin] PATH_INFO: `/v3/projects/5c62d063a0ca4a11ae02b5f8f2ef8be9/users/7962bbb5fbab4f2593a3d3d2c4140e79/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:58.738366 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dc48e30-1474-4215-8c20-2c0bd454df3e admin admin] RBAC: Authorizing `identity:create_grant(project_id=5c62d063a0ca4a11ae02b5f8f2ef8be9, user_id=7962bbb5fbab4f2593a3d3d2c4140e79, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:04:58.740925 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dc48e30-1474-4215-8c20-2c0bd454df3e admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:04:58.753090 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1426/2861] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 01:04:58 2022] PUT /identity/v3/projects/5c62d063a0ca4a11ae02b5f8f2ef8be9/users/7962bbb5fbab4f2593a3d3d2c4140e79/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 01:04:58.756972 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567d2ce5-d598-43aa-ae80-c2af7a810d22 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:04:58.767480 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567d2ce5-d598-43aa-ae80-c2af7a810d22 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:04:58.824320 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fedfd68-1b26-4d70-aa1f-364f90c241f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:04:58.824925 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fedfd68-1b26-4d70-aa1f-364f90c241f3 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:58.825204 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fedfd68-1b26-4d70-aa1f-364f90c241f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:58.826020 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fedfd68-1b26-4d70-aa1f-364f90c241f3 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:58.826818 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fedfd68-1b26-4d70-aa1f-364f90c241f3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:04:58.828434 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fedfd68-1b26-4d70-aa1f-364f90c241f3 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:04:58.838006 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1436/2862] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 01:04:58 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 01:04:58.843247 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dc48e30-1474-4215-8c20-2c0bd454df3e admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:04:58.853293 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dc48e30-1474-4215-8c20-2c0bd454df3e admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:04:58.857896 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f29edab-a523-404f-91eb-83cc82c5df0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:04:58.858538 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f29edab-a523-404f-91eb-83cc82c5df0d admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:58.858844 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f29edab-a523-404f-91eb-83cc82c5df0d admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:58.859120 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f29edab-a523-404f-91eb-83cc82c5df0d admin admin] PATH_INFO: `/v3/projects/5c62d063a0ca4a11ae02b5f8f2ef8be9/users/7962bbb5fbab4f2593a3d3d2c4140e79/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:58.861048 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f29edab-a523-404f-91eb-83cc82c5df0d admin admin] RBAC: Authorizing `identity:create_grant(project_id=5c62d063a0ca4a11ae02b5f8f2ef8be9, user_id=7962bbb5fbab4f2593a3d3d2c4140e79, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:04:58.863706 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f29edab-a523-404f-91eb-83cc82c5df0d admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:04:58.873773 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1427/2863] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 01:04:58 2022] PUT /identity/v3/projects/5c62d063a0ca4a11ae02b5f8f2ef8be9/users/7962bbb5fbab4f2593a3d3d2c4140e79/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 01:04:58.883937 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5aad41-e8b4-40f2-a0ba-997e0ce9de4f None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:58.884228 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5aad41-e8b4-40f2-a0ba-997e0ce9de4f None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:58.884504 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5aad41-e8b4-40f2-a0ba-997e0ce9de4f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:58.912791 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-2b5aad41-e8b4-40f2-a0ba-997e0ce9de4f None None] MFA Rules not processed for user `7962bbb5fbab4f2593a3d3d2c4140e79`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:04:58.913987 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-2b5aad41-e8b4-40f2-a0ba-997e0ce9de4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:04:58.963741 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:04:58.983705 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1437/2864] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 01:04:58 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 01:04:58.994215 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f29edab-a523-404f-91eb-83cc82c5df0d admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:04:58.996296 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-4f29edab-a523-404f-91eb-83cc82c5df0d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:04:59.053893 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f29edab-a523-404f-91eb-83cc82c5df0d admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:04:59.057582 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c9e0b4-f958-4e55-987a-e81bb978b305 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:04:59.058177 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c9e0b4-f958-4e55-987a-e81bb978b305 service cinder] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:59.058455 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c9e0b4-f958-4e55-987a-e81bb978b305 service cinder] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:59.058729 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c9e0b4-f958-4e55-987a-e81bb978b305 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:59.069292 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60c9e0b4-f958-4e55-987a-e81bb978b305 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:04:59.070724 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60c9e0b4-f958-4e55-987a-e81bb978b305 service cinder] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:04:59.080256 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1428/2865] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 01:04:58 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 01:04:59.447312 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b5aad41-e8b4-40f2-a0ba-997e0ce9de4f None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:04:59.508856 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b5aad41-e8b4-40f2-a0ba-997e0ce9de4f None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:04:59.513927 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b677a6-d2a6-4f51-bc1d-61bdd53d131b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:04:59.514551 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b677a6-d2a6-4f51-bc1d-61bdd53d131b admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:59.514859 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b677a6-d2a6-4f51-bc1d-61bdd53d131b admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:59.515137 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b677a6-d2a6-4f51-bc1d-61bdd53d131b admin admin] PATH_INFO: `/v3/users/cebe9e04cb2848f39e99e7195fcf5dc1` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:59.516261 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b677a6-d2a6-4f51-bc1d-61bdd53d131b admin admin] RBAC: Authorizing `identity:delete_user(user_id=cebe9e04cb2848f39e99e7195fcf5dc1)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:04:59.518211 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b677a6-d2a6-4f51-bc1d-61bdd53d131b admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:04:59.564943 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-e3b677a6-d2a6-4f51-bc1d-61bdd53d131b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cebe9e04cb2848f39e99e7195fcf5dc1'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:04:59.565589 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-e3b677a6-d2a6-4f51-bc1d-61bdd53d131b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cebe9e04cb2848f39e99e7195fcf5dc1'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:04:59.566195 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-e3b677a6-d2a6-4f51-bc1d-61bdd53d131b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cebe9e04cb2848f39e99e7195fcf5dc1'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:04:59.573040 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-e3b677a6-d2a6-4f51-bc1d-61bdd53d131b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cebe9e04cb2848f39e99e7195fcf5dc1'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:04:59.585262 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-e3b677a6-d2a6-4f51-bc1d-61bdd53d131b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cebe9e04cb2848f39e99e7195fcf5dc1'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:04:59.598069 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1438/2866] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 01:04:59 2022] DELETE /identity/v3/users/cebe9e04cb2848f39e99e7195fcf5dc1 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 01:04:59.602738 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5679117e-ffa1-48fe-86da-f85cbe70293b None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:59.603087 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5679117e-ffa1-48fe-86da-f85cbe70293b None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:59.603362 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5679117e-ffa1-48fe-86da-f85cbe70293b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:59.622081 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-5679117e-ffa1-48fe-86da-f85cbe70293b None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:04:59.623581 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-5679117e-ffa1-48fe-86da-f85cbe70293b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:04:59.681565 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1429/2867] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 01:04:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 01:04:59.691268 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b677a6-d2a6-4f51-bc1d-61bdd53d131b admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:04:59.692652 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-e3b677a6-d2a6-4f51-bc1d-61bdd53d131b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:04:59.749232 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b677a6-d2a6-4f51-bc1d-61bdd53d131b admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:04:59.752922 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41207b3a-5d5f-456b-ad7a-efdfc9d8d3a1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:04:59.753551 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41207b3a-5d5f-456b-ad7a-efdfc9d8d3a1 service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:04:59.753827 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41207b3a-5d5f-456b-ad7a-efdfc9d8d3a1 service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:04:59.754096 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41207b3a-5d5f-456b-ad7a-efdfc9d8d3a1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:04:59.763684 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41207b3a-5d5f-456b-ad7a-efdfc9d8d3a1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:04:59.765040 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41207b3a-5d5f-456b-ad7a-efdfc9d8d3a1 service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:04:59.774332 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1439/2868] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 01:04:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 01:05:00.038322 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5679117e-ffa1-48fe-86da-f85cbe70293b None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:05:00.039843 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-5679117e-ffa1-48fe-86da-f85cbe70293b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:05:00.052921 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5679117e-ffa1-48fe-86da-f85cbe70293b None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:05:00.057467 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a0939a1-1d4c-4a29-b790-6d02970d1813 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:05:00.058139 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0939a1-1d4c-4a29-b790-6d02970d1813 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:00.058417 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0939a1-1d4c-4a29-b790-6d02970d1813 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:00.058691 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0939a1-1d4c-4a29-b790-6d02970d1813 admin admin] PATH_INFO: `/v3/projects/a0a17562cbbf41909c74b6319ac0898b` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:00.060487 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a0939a1-1d4c-4a29-b790-6d02970d1813 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a0a17562cbbf41909c74b6319ac0898b)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:05:00.062882 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a0939a1-1d4c-4a29-b790-6d02970d1813 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:05:00.079794 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:05:00.112065 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-6a0939a1-1d4c-4a29-b790-6d02970d1813 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a0a17562cbbf41909c74b6319ac0898b', 'request_id': 'req-6a0939a1-1d4c-4a29-b790-6d02970d1813'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:05:00.118144 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-6a0939a1-1d4c-4a29-b790-6d02970d1813 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a0a17562cbbf41909c74b6319ac0898b', 'request_id': 'req-6a0939a1-1d4c-4a29-b790-6d02970d1813'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:05:00.124396 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-6a0939a1-1d4c-4a29-b790-6d02970d1813 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a0a17562cbbf41909c74b6319ac0898b', 'request_id': 'req-6a0939a1-1d4c-4a29-b790-6d02970d1813'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:05:00.136627 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-6a0939a1-1d4c-4a29-b790-6d02970d1813 admin admin] The token cache is being invalidate because project a0a17562cbbf41909c74b6319ac0898b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87923) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 01:05:00.136914 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-6a0939a1-1d4c-4a29-b790-6d02970d1813 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:05:00.137526 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-6a0939a1-1d4c-4a29-b790-6d02970d1813 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:05:00.140131 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1430/2869] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 01:05:00 2022] DELETE /identity/v3/projects/a0a17562cbbf41909c74b6319ac0898b => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 01:05:00.144133 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41207b3a-5d5f-456b-ad7a-efdfc9d8d3a1 service neutron] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:05:00.154062 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41207b3a-5d5f-456b-ad7a-efdfc9d8d3a1 service neutron] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:05:00.162062 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:05:00.211388 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1581da84-e145-4016-b9ea-db0344142aba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:05:00.212003 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1581da84-e145-4016-b9ea-db0344142aba admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:00.212277 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1581da84-e145-4016-b9ea-db0344142aba admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:00.212544 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1581da84-e145-4016-b9ea-db0344142aba admin admin] PATH_INFO: `/v3/users/7962bbb5fbab4f2593a3d3d2c4140e79` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:00.213641 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1581da84-e145-4016-b9ea-db0344142aba admin admin] RBAC: Authorizing `identity:delete_user(user_id=7962bbb5fbab4f2593a3d3d2c4140e79)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:05:00.215573 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1581da84-e145-4016-b9ea-db0344142aba admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:05:00.259901 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-1581da84-e145-4016-b9ea-db0344142aba admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7962bbb5fbab4f2593a3d3d2c4140e79'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:05:00.260547 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-1581da84-e145-4016-b9ea-db0344142aba admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7962bbb5fbab4f2593a3d3d2c4140e79'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:05:00.261139 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-1581da84-e145-4016-b9ea-db0344142aba admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7962bbb5fbab4f2593a3d3d2c4140e79'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:05:00.268532 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-1581da84-e145-4016-b9ea-db0344142aba admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7962bbb5fbab4f2593a3d3d2c4140e79'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:05:00.280166 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-1581da84-e145-4016-b9ea-db0344142aba admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7962bbb5fbab4f2593a3d3d2c4140e79'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:05:00.292601 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1440/2870] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 01:05:00 2022] DELETE /identity/v3/users/7962bbb5fbab4f2593a3d3d2c4140e79 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 01:05:00.561954 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a0939a1-1d4c-4a29-b790-6d02970d1813 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:05:00.563397 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-6a0939a1-1d4c-4a29-b790-6d02970d1813 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:05:00.576112 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a0939a1-1d4c-4a29-b790-6d02970d1813 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:05:00.634010 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-276d7a3a-a7d3-4b07-a01e-92939fb19d29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:05:00.634637 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276d7a3a-a7d3-4b07-a01e-92939fb19d29 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:00.634952 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276d7a3a-a7d3-4b07-a01e-92939fb19d29 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:00.635226 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276d7a3a-a7d3-4b07-a01e-92939fb19d29 admin admin] PATH_INFO: `/v3/projects/5c62d063a0ca4a11ae02b5f8f2ef8be9` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:00.636336 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-276d7a3a-a7d3-4b07-a01e-92939fb19d29 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5c62d063a0ca4a11ae02b5f8f2ef8be9)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:05:00.638267 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-276d7a3a-a7d3-4b07-a01e-92939fb19d29 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:05:00.684268 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-276d7a3a-a7d3-4b07-a01e-92939fb19d29 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5c62d063a0ca4a11ae02b5f8f2ef8be9', 'request_id': 'req-276d7a3a-a7d3-4b07-a01e-92939fb19d29'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:05:00.690307 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-276d7a3a-a7d3-4b07-a01e-92939fb19d29 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5c62d063a0ca4a11ae02b5f8f2ef8be9', 'request_id': 'req-276d7a3a-a7d3-4b07-a01e-92939fb19d29'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:05:00.695882 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-276d7a3a-a7d3-4b07-a01e-92939fb19d29 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5c62d063a0ca4a11ae02b5f8f2ef8be9', 'request_id': 'req-276d7a3a-a7d3-4b07-a01e-92939fb19d29'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:05:00.707572 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-276d7a3a-a7d3-4b07-a01e-92939fb19d29 admin admin] The token cache is being invalidate because project 5c62d063a0ca4a11ae02b5f8f2ef8be9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87923) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 01:05:00.707858 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-276d7a3a-a7d3-4b07-a01e-92939fb19d29 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:05:00.708468 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-276d7a3a-a7d3-4b07-a01e-92939fb19d29 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:05:00.709929 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1431/2871] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 01:05:00 2022] DELETE /identity/v3/projects/5c62d063a0ca4a11ae02b5f8f2ef8be9 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 01:05:00.714838 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15269a2d-1945-4f7b-acd6-9ed6342aa030 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:00.715127 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15269a2d-1945-4f7b-acd6-9ed6342aa030 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:00.715398 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15269a2d-1945-4f7b-acd6-9ed6342aa030 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:00.736238 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-15269a2d-1945-4f7b-acd6-9ed6342aa030 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:05:00.737445 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-15269a2d-1945-4f7b-acd6-9ed6342aa030 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:05:00.795979 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1441/2872] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 01:05:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 01:05:00.803681 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5470b22-1fc6-4bcb-bfbe-b3831fe6c610 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:00.803966 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5470b22-1fc6-4bcb-bfbe-b3831fe6c610 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:00.804287 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5470b22-1fc6-4bcb-bfbe-b3831fe6c610 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:00.823512 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-c5470b22-1fc6-4bcb-bfbe-b3831fe6c610 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:05:00.824683 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-c5470b22-1fc6-4bcb-bfbe-b3831fe6c610 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:05:00.832738 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1432/2873] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 01:05:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 01:05:00.836857 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15269a2d-1945-4f7b-acd6-9ed6342aa030 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:05:00.847541 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15269a2d-1945-4f7b-acd6-9ed6342aa030 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:05:00.852101 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f69483-0b38-4820-9403-b0abd47d77f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:05:00.852689 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f69483-0b38-4820-9403-b0abd47d77f9 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:00.852964 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f69483-0b38-4820-9403-b0abd47d77f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:00.853234 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f69483-0b38-4820-9403-b0abd47d77f9 admin admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:00.853875 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3f69483-0b38-4820-9403-b0abd47d77f9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 01:05:00.854174 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3f69483-0b38-4820-9403-b0abd47d77f9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:05:00.855793 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3f69483-0b38-4820-9403-b0abd47d77f9 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:05:00.867455 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1442/2874] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 01:05:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 01:05:00.871535 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5470b22-1fc6-4bcb-bfbe-b3831fe6c610 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:05:00.880646 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5470b22-1fc6-4bcb-bfbe-b3831fe6c610 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:05:00.885121 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d70707a5-094b-4264-ba5d-fb5d74a7bcdf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:05:00.885698 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70707a5-094b-4264-ba5d-fb5d74a7bcdf admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:00.885974 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70707a5-094b-4264-ba5d-fb5d74a7bcdf admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:00.886239 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70707a5-094b-4264-ba5d-fb5d74a7bcdf admin admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:00.886889 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d70707a5-094b-4264-ba5d-fb5d74a7bcdf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:05:00.888865 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d70707a5-094b-4264-ba5d-fb5d74a7bcdf admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:05:00.916626 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1433/2875] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 01:05:00 2022] POST /identity/v3/projects => generated 383 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 01:05:00.923028 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f69483-0b38-4820-9403-b0abd47d77f9 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:05:00.932803 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f69483-0b38-4820-9403-b0abd47d77f9 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:05:00.944923 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:05:00.989728 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cafc977e-2f46-429a-b00b-295da714f250 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:05:00.990327 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafc977e-2f46-429a-b00b-295da714f250 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:00.990605 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafc977e-2f46-429a-b00b-295da714f250 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:00.990916 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafc977e-2f46-429a-b00b-295da714f250 admin admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:00.991540 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cafc977e-2f46-429a-b00b-295da714f250 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:05:00.993441 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cafc977e-2f46-429a-b00b-295da714f250 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:05:01.014620 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1443/2876] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 01:05:00 2022] POST /identity/v3/users => generated 292 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 01:05:01.018746 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d70707a5-094b-4264-ba5d-fb5d74a7bcdf admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:05:01.025551 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:05:01.030300 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d70707a5-094b-4264-ba5d-fb5d74a7bcdf admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:05:01.036327 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd1e2833-e8c7-4991-8626-d85d794c8b83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:05:01.037140 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1e2833-e8c7-4991-8626-d85d794c8b83 admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:01.037618 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1e2833-e8c7-4991-8626-d85d794c8b83 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:01.038097 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1e2833-e8c7-4991-8626-d85d794c8b83 admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:01.038825 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd1e2833-e8c7-4991-8626-d85d794c8b83 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:05:01.040477 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd1e2833-e8c7-4991-8626-d85d794c8b83 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:05:01.050500 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1434/2877] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 01:05:01 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 01:05:01.054486 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cafc977e-2f46-429a-b00b-295da714f250 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:05:01.064456 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cafc977e-2f46-429a-b00b-295da714f250 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:05:01.069026 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a32dfa-5aac-4343-82f8-c792ab1ad0d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:05:01.069612 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a32dfa-5aac-4343-82f8-c792ab1ad0d5 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:01.069891 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a32dfa-5aac-4343-82f8-c792ab1ad0d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:01.070162 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a32dfa-5aac-4343-82f8-c792ab1ad0d5 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:01.070816 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5a32dfa-5aac-4343-82f8-c792ab1ad0d5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:05:01.072418 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5a32dfa-5aac-4343-82f8-c792ab1ad0d5 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:05:01.083737 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1444/2878] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 01:05:01 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 01:05:01.087895 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd1e2833-e8c7-4991-8626-d85d794c8b83 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:05:01.098634 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd1e2833-e8c7-4991-8626-d85d794c8b83 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:05:01.103316 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23cc1477-f8a9-41c3-a85c-0aeca984b888 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:05:01.104056 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23cc1477-f8a9-41c3-a85c-0aeca984b888 admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:01.104333 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23cc1477-f8a9-41c3-a85c-0aeca984b888 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:01.105072 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23cc1477-f8a9-41c3-a85c-0aeca984b888 admin admin] PATH_INFO: `/v3/projects/914d8ed513b94e8a951b9270c134f064/users/5938916a893940368eeb0159d99de989/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:01.119328 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23cc1477-f8a9-41c3-a85c-0aeca984b888 admin admin] RBAC: Authorizing `identity:create_grant(project_id=914d8ed513b94e8a951b9270c134f064, user_id=5938916a893940368eeb0159d99de989, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:05:01.121947 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23cc1477-f8a9-41c3-a85c-0aeca984b888 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:05:01.134893 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1435/2879] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 01:05:01 2022] PUT /identity/v3/projects/914d8ed513b94e8a951b9270c134f064/users/5938916a893940368eeb0159d99de989/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 01:05:01.198409 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13805b9a-6a72-44ef-8ef9-3f5f36fde0e9 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:01.198694 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13805b9a-6a72-44ef-8ef9-3f5f36fde0e9 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:01.198988 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13805b9a-6a72-44ef-8ef9-3f5f36fde0e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:01.227102 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-13805b9a-6a72-44ef-8ef9-3f5f36fde0e9 None None] MFA Rules not processed for user `5938916a893940368eeb0159d99de989`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:05:01.228307 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-13805b9a-6a72-44ef-8ef9-3f5f36fde0e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:05:01.291263 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1445/2880] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 01:05:01 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 01:05:01.295904 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23cc1477-f8a9-41c3-a85c-0aeca984b888 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:05:01.357115 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23cc1477-f8a9-41c3-a85c-0aeca984b888 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:05:01.361687 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9804a8b1-dd2d-45a0-9018-407de0c22f98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:05:01.362288 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9804a8b1-dd2d-45a0-9018-407de0c22f98 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:01.362594 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9804a8b1-dd2d-45a0-9018-407de0c22f98 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:01.362901 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9804a8b1-dd2d-45a0-9018-407de0c22f98 admin admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:01.363535 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9804a8b1-dd2d-45a0-9018-407de0c22f98 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:05:01.365430 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9804a8b1-dd2d-45a0-9018-407de0c22f98 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:05:01.393209 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1436/2881] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 01:05:01 2022] POST /identity/v3/projects => generated 381 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 01:05:01.397302 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13805b9a-6a72-44ef-8ef9-3f5f36fde0e9 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:05:01.407931 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13805b9a-6a72-44ef-8ef9-3f5f36fde0e9 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:05:01.463555 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7308bd72-1300-447c-828c-c45880e40d24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:05:01.464156 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7308bd72-1300-447c-828c-c45880e40d24 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:01.464432 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7308bd72-1300-447c-828c-c45880e40d24 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:01.464718 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7308bd72-1300-447c-828c-c45880e40d24 admin admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:01.465371 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7308bd72-1300-447c-828c-c45880e40d24 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:05:01.467518 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7308bd72-1300-447c-828c-c45880e40d24 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:05:01.488200 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1446/2882] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 01:05:01 2022] POST /identity/v3/users => generated 297 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 01:05:01.492302 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9804a8b1-dd2d-45a0-9018-407de0c22f98 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:05:01.503171 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9804a8b1-dd2d-45a0-9018-407de0c22f98 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:05:01.507767 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2237b072-ee28-4a10-8898-5df8e7948bee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:05:01.508447 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2237b072-ee28-4a10-8898-5df8e7948bee admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:01.508728 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2237b072-ee28-4a10-8898-5df8e7948bee admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:01.509001 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2237b072-ee28-4a10-8898-5df8e7948bee admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:01.509646 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2237b072-ee28-4a10-8898-5df8e7948bee admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:05:01.511283 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2237b072-ee28-4a10-8898-5df8e7948bee admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:05:01.522278 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1437/2883] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 01:05:01 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 01:05:01.526201 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7308bd72-1300-447c-828c-c45880e40d24 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:05:01.536465 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7308bd72-1300-447c-828c-c45880e40d24 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:05:01.541176 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc923bb-4a52-4c75-80f5-fd600952c46b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:05:01.541802 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc923bb-4a52-4c75-80f5-fd600952c46b admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:01.542078 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc923bb-4a52-4c75-80f5-fd600952c46b admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:01.542351 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc923bb-4a52-4c75-80f5-fd600952c46b admin admin] PATH_INFO: `/v3/projects/3f4fafa364cc46d699a68f1056b101b1/users/5fd2f49c7b5246c6b03a90ac1025fe8e/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:01.554520 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:05:01.557706 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bc923bb-4a52-4c75-80f5-fd600952c46b admin admin] RBAC: Authorizing `identity:create_grant(project_id=3f4fafa364cc46d699a68f1056b101b1, user_id=5fd2f49c7b5246c6b03a90ac1025fe8e, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:05:01.561429 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bc923bb-4a52-4c75-80f5-fd600952c46b admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:05:01.575017 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1447/2884] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 01:05:01 2022] PUT /identity/v3/projects/3f4fafa364cc46d699a68f1056b101b1/users/5fd2f49c7b5246c6b03a90ac1025fe8e/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 01:05:01.579534 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2237b072-ee28-4a10-8898-5df8e7948bee admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:05:01.589378 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2237b072-ee28-4a10-8898-5df8e7948bee admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:05:01.646281 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab97b88-884f-4e2c-bc33-710b7c016f3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:05:01.646830 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab97b88-884f-4e2c-bc33-710b7c016f3d admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:01.647111 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab97b88-884f-4e2c-bc33-710b7c016f3d admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:01.647376 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab97b88-884f-4e2c-bc33-710b7c016f3d admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:01.648005 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ab97b88-884f-4e2c-bc33-710b7c016f3d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:05:01.649581 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ab97b88-884f-4e2c-bc33-710b7c016f3d admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:05:01.659479 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1438/2885] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 01:05:01 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 01:05:01.663750 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc923bb-4a52-4c75-80f5-fd600952c46b admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:05:01.673755 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc923bb-4a52-4c75-80f5-fd600952c46b admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:05:01.678275 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b4acab-2348-4649-9f21-abe3577a68d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:05:01.678944 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b4acab-2348-4649-9f21-abe3577a68d3 admin admin] REQUEST_METHOD: `PUT` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:01.679222 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b4acab-2348-4649-9f21-abe3577a68d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:01.679494 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b4acab-2348-4649-9f21-abe3577a68d3 admin admin] PATH_INFO: `/v3/projects/3f4fafa364cc46d699a68f1056b101b1/users/5fd2f49c7b5246c6b03a90ac1025fe8e/roles/7d9213c1187c4b5182bf8132cf23ea19` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:01.681386 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b4acab-2348-4649-9f21-abe3577a68d3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3f4fafa364cc46d699a68f1056b101b1, user_id=5fd2f49c7b5246c6b03a90ac1025fe8e, role_id=7d9213c1187c4b5182bf8132cf23ea19)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:05:01.684066 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b4acab-2348-4649-9f21-abe3577a68d3 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:05:01.694427 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1448/2886] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 01:05:01 2022] PUT /identity/v3/projects/3f4fafa364cc46d699a68f1056b101b1/users/5fd2f49c7b5246c6b03a90ac1025fe8e/roles/7d9213c1187c4b5182bf8132cf23ea19 => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 01:05:01.705428 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d823e8-02df-401e-97b2-6a4bc0e6dc94 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:01.705712 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d823e8-02df-401e-97b2-6a4bc0e6dc94 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:01.705985 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d823e8-02df-401e-97b2-6a4bc0e6dc94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:01.732735 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-10d823e8-02df-401e-97b2-6a4bc0e6dc94 None None] MFA Rules not processed for user `5fd2f49c7b5246c6b03a90ac1025fe8e`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:05:01.733920 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-10d823e8-02df-401e-97b2-6a4bc0e6dc94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:05:01.762184 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:05:01.802480 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1439/2887] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 01:05:01 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 01:05:01.814866 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b4acab-2348-4649-9f21-abe3577a68d3 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:05:01.816579 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-24b4acab-2348-4649-9f21-abe3577a68d3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:05:01.875726 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b4acab-2348-4649-9f21-abe3577a68d3 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:05:01.879415 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dfc26b7-7fde-4990-a799-65e7a3fc4b84 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:05:01.880018 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfc26b7-7fde-4990-a799-65e7a3fc4b84 service cinder] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:01.880298 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfc26b7-7fde-4990-a799-65e7a3fc4b84 service cinder] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:01.880571 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfc26b7-7fde-4990-a799-65e7a3fc4b84 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:01.889565 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dfc26b7-7fde-4990-a799-65e7a3fc4b84 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:05:01.891020 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dfc26b7-7fde-4990-a799-65e7a3fc4b84 service cinder] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:05:01.900433 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1449/2888] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 01:05:01 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 01:05:01.973282 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10d823e8-02df-401e-97b2-6a4bc0e6dc94 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:05:01.983576 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10d823e8-02df-401e-97b2-6a4bc0e6dc94 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:05:01.987124 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4826b6c-d12d-46da-be3e-71fa7ddbee0c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:05:01.987709 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4826b6c-d12d-46da-be3e-71fa7ddbee0c service cinder] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:01.987977 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4826b6c-d12d-46da-be3e-71fa7ddbee0c service cinder] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:01.988240 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4826b6c-d12d-46da-be3e-71fa7ddbee0c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:02.043678 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4826b6c-d12d-46da-be3e-71fa7ddbee0c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:05:02.045779 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4826b6c-d12d-46da-be3e-71fa7ddbee0c service cinder] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:05:02.054651 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1440/2889] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 01:05:01 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 01:05:02.317968 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dfc26b7-7fde-4990-a799-65e7a3fc4b84 service cinder] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:05:02.378924 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dfc26b7-7fde-4990-a799-65e7a3fc4b84 service cinder] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:05:02.383479 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c164f4e-585a-49bc-9b15-7b8a2ce35be5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:05:02.384097 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c164f4e-585a-49bc-9b15-7b8a2ce35be5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:02.384370 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c164f4e-585a-49bc-9b15-7b8a2ce35be5 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:02.384637 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c164f4e-585a-49bc-9b15-7b8a2ce35be5 admin admin] PATH_INFO: `/v3/users/5938916a893940368eeb0159d99de989` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:02.385730 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c164f4e-585a-49bc-9b15-7b8a2ce35be5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5938916a893940368eeb0159d99de989)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:05:02.387668 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c164f4e-585a-49bc-9b15-7b8a2ce35be5 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:05:02.435853 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-6c164f4e-585a-49bc-9b15-7b8a2ce35be5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5938916a893940368eeb0159d99de989'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:05:02.435853 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-6c164f4e-585a-49bc-9b15-7b8a2ce35be5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5938916a893940368eeb0159d99de989'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:05:02.436354 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-6c164f4e-585a-49bc-9b15-7b8a2ce35be5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5938916a893940368eeb0159d99de989'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:05:02.443497 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-6c164f4e-585a-49bc-9b15-7b8a2ce35be5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5938916a893940368eeb0159d99de989'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:05:02.455312 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-6c164f4e-585a-49bc-9b15-7b8a2ce35be5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5938916a893940368eeb0159d99de989'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:05:02.463838 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=87,keystone:INSERT=3,keystone:DELETE=10 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:05:02.469357 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1450/2890] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 01:05:02 2022] DELETE /identity/v3/users/5938916a893940368eeb0159d99de989 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 01:05:02.473855 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e97ceb6-616a-4e37-a2a1-49e5fbe78698 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:02.474137 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e97ceb6-616a-4e37-a2a1-49e5fbe78698 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:02.474407 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e97ceb6-616a-4e37-a2a1-49e5fbe78698 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:02.493640 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-8e97ceb6-616a-4e37-a2a1-49e5fbe78698 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:05:02.494840 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-8e97ceb6-616a-4e37-a2a1-49e5fbe78698 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:05:02.552154 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1441/2891] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 01:05:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 01:05:02.561916 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c164f4e-585a-49bc-9b15-7b8a2ce35be5 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:05:02.563347 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-6c164f4e-585a-49bc-9b15-7b8a2ce35be5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:05:02.621638 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c164f4e-585a-49bc-9b15-7b8a2ce35be5 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:05:02.625501 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-315434a6-896a-48b3-b361-ecb621a137d0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:05:02.626094 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315434a6-896a-48b3-b361-ecb621a137d0 service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:02.626366 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315434a6-896a-48b3-b361-ecb621a137d0 service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:02.626630 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315434a6-896a-48b3-b361-ecb621a137d0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:02.635386 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-315434a6-896a-48b3-b361-ecb621a137d0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:05:02.636794 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-315434a6-896a-48b3-b361-ecb621a137d0 service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:05:02.647108 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1451/2892] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 01:05:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 01:05:02.911699 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e97ceb6-616a-4e37-a2a1-49e5fbe78698 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:05:02.913235 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-8e97ceb6-616a-4e37-a2a1-49e5fbe78698 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:05:02.925633 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e97ceb6-616a-4e37-a2a1-49e5fbe78698 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:05:02.930176 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9156b608-1481-4db9-a82d-11029dd41d73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:05:02.930866 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9156b608-1481-4db9-a82d-11029dd41d73 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:02.931139 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9156b608-1481-4db9-a82d-11029dd41d73 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:02.931410 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9156b608-1481-4db9-a82d-11029dd41d73 admin admin] PATH_INFO: `/v3/projects/914d8ed513b94e8a951b9270c134f064` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:02.932564 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9156b608-1481-4db9-a82d-11029dd41d73 admin admin] RBAC: Authorizing `identity:delete_project(project_id=914d8ed513b94e8a951b9270c134f064)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:05:02.934513 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9156b608-1481-4db9-a82d-11029dd41d73 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:05:02.981546 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-9156b608-1481-4db9-a82d-11029dd41d73 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '914d8ed513b94e8a951b9270c134f064', 'request_id': 'req-9156b608-1481-4db9-a82d-11029dd41d73'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:05:02.987697 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-9156b608-1481-4db9-a82d-11029dd41d73 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '914d8ed513b94e8a951b9270c134f064', 'request_id': 'req-9156b608-1481-4db9-a82d-11029dd41d73'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:05:02.991546 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:05:02.994105 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-9156b608-1481-4db9-a82d-11029dd41d73 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '914d8ed513b94e8a951b9270c134f064', 'request_id': 'req-9156b608-1481-4db9-a82d-11029dd41d73'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:05:03.007224 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-9156b608-1481-4db9-a82d-11029dd41d73 admin admin] The token cache is being invalidate because project 914d8ed513b94e8a951b9270c134f064 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87923) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 01:05:03.007513 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-9156b608-1481-4db9-a82d-11029dd41d73 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:05:03.008133 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-9156b608-1481-4db9-a82d-11029dd41d73 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:05:03.009626 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1442/2893] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 01:05:02 2022] DELETE /identity/v3/projects/914d8ed513b94e8a951b9270c134f064 => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 01:05:03.014469 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-315434a6-896a-48b3-b361-ecb621a137d0 service neutron] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:05:03.024032 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-315434a6-896a-48b3-b361-ecb621a137d0 service neutron] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:05:03.079471 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a521d6e-81f4-48b5-9d76-eff3302c043b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:05:03.080107 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a521d6e-81f4-48b5-9d76-eff3302c043b admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:03.080382 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a521d6e-81f4-48b5-9d76-eff3302c043b admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:03.080654 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a521d6e-81f4-48b5-9d76-eff3302c043b admin admin] PATH_INFO: `/v3/users/5fd2f49c7b5246c6b03a90ac1025fe8e` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:03.081796 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a521d6e-81f4-48b5-9d76-eff3302c043b admin admin] RBAC: Authorizing `identity:delete_user(user_id=5fd2f49c7b5246c6b03a90ac1025fe8e)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:05:03.083739 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a521d6e-81f4-48b5-9d76-eff3302c043b admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:05:03.129785 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-4a521d6e-81f4-48b5-9d76-eff3302c043b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5fd2f49c7b5246c6b03a90ac1025fe8e'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:05:03.130436 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-4a521d6e-81f4-48b5-9d76-eff3302c043b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5fd2f49c7b5246c6b03a90ac1025fe8e'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:05:03.131077 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-4a521d6e-81f4-48b5-9d76-eff3302c043b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5fd2f49c7b5246c6b03a90ac1025fe8e'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:05:03.138614 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-4a521d6e-81f4-48b5-9d76-eff3302c043b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5fd2f49c7b5246c6b03a90ac1025fe8e'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:05:03.150827 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-4a521d6e-81f4-48b5-9d76-eff3302c043b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5fd2f49c7b5246c6b03a90ac1025fe8e'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:05:03.163566 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1452/2894] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 01:05:03 2022] DELETE /identity/v3/users/5fd2f49c7b5246c6b03a90ac1025fe8e => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 01:05:03.436709 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9156b608-1481-4db9-a82d-11029dd41d73 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:05:03.438180 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-9156b608-1481-4db9-a82d-11029dd41d73 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:05:03.457365 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9156b608-1481-4db9-a82d-11029dd41d73 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:05:03.514303 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af12daf5-8b0c-4590-97dc-f276e893ffee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:05:03.514898 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af12daf5-8b0c-4590-97dc-f276e893ffee admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:03.515182 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af12daf5-8b0c-4590-97dc-f276e893ffee admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:03.515453 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af12daf5-8b0c-4590-97dc-f276e893ffee admin admin] PATH_INFO: `/v3/projects/3f4fafa364cc46d699a68f1056b101b1` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:03.516610 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af12daf5-8b0c-4590-97dc-f276e893ffee admin admin] RBAC: Authorizing `identity:delete_project(project_id=3f4fafa364cc46d699a68f1056b101b1)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:05:03.518623 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af12daf5-8b0c-4590-97dc-f276e893ffee admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:05:03.565739 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-af12daf5-8b0c-4590-97dc-f276e893ffee admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3f4fafa364cc46d699a68f1056b101b1', 'request_id': 'req-af12daf5-8b0c-4590-97dc-f276e893ffee'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:05:03.571418 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-af12daf5-8b0c-4590-97dc-f276e893ffee admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3f4fafa364cc46d699a68f1056b101b1', 'request_id': 'req-af12daf5-8b0c-4590-97dc-f276e893ffee'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:05:03.577313 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-af12daf5-8b0c-4590-97dc-f276e893ffee admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3f4fafa364cc46d699a68f1056b101b1', 'request_id': 'req-af12daf5-8b0c-4590-97dc-f276e893ffee'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:05:03.588900 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-af12daf5-8b0c-4590-97dc-f276e893ffee admin admin] The token cache is being invalidate because project 3f4fafa364cc46d699a68f1056b101b1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87923) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 01:05:03.589181 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-af12daf5-8b0c-4590-97dc-f276e893ffee admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:05:03.589776 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-af12daf5-8b0c-4590-97dc-f276e893ffee admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:05:03.591258 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1443/2895] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 01:05:03 2022] DELETE /identity/v3/projects/3f4fafa364cc46d699a68f1056b101b1 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 01:05:03.596637 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c110a989-5d5c-4711-ace3-55db7eae0a55 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:03.596927 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c110a989-5d5c-4711-ace3-55db7eae0a55 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:03.597203 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c110a989-5d5c-4711-ace3-55db7eae0a55 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:03.620143 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-c110a989-5d5c-4711-ace3-55db7eae0a55 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:05:03.621770 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-c110a989-5d5c-4711-ace3-55db7eae0a55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:05:03.639862 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:05:03.683863 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1453/2896] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 01:05:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 01:05:03.692943 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7413cfcc-efd4-47c2-ae08-91bb71cb9b23 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:03.693222 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7413cfcc-efd4-47c2-ae08-91bb71cb9b23 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:03.693487 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7413cfcc-efd4-47c2-ae08-91bb71cb9b23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:03.713611 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-7413cfcc-efd4-47c2-ae08-91bb71cb9b23 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:05:03.714813 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-7413cfcc-efd4-47c2-ae08-91bb71cb9b23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:05:03.723138 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1444/2897] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 01:05:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 01:05:03.727259 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c110a989-5d5c-4711-ace3-55db7eae0a55 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:05:03.737613 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c110a989-5d5c-4711-ace3-55db7eae0a55 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:05:03.742087 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18e25d4-c56d-4be7-93d6-033a75ffb813 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:05:03.742675 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18e25d4-c56d-4be7-93d6-033a75ffb813 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:03.742982 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18e25d4-c56d-4be7-93d6-033a75ffb813 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:03.743249 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18e25d4-c56d-4be7-93d6-033a75ffb813 admin admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:03.743888 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d18e25d4-c56d-4be7-93d6-033a75ffb813 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 01:05:03.744184 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d18e25d4-c56d-4be7-93d6-033a75ffb813 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:05:03.745760 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d18e25d4-c56d-4be7-93d6-033a75ffb813 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:05:03.762286 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1454/2898] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 01:05:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 01:05:03.763526 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7413cfcc-efd4-47c2-ae08-91bb71cb9b23 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:05:03.772805 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7413cfcc-efd4-47c2-ae08-91bb71cb9b23 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:05:03.777332 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49543e01-03a2-44ae-98e1-98537991f948 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:05:03.777916 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49543e01-03a2-44ae-98e1-98537991f948 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:03.778187 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49543e01-03a2-44ae-98e1-98537991f948 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:03.778447 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49543e01-03a2-44ae-98e1-98537991f948 admin admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:03.779976 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49543e01-03a2-44ae-98e1-98537991f948 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:05:03.781848 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49543e01-03a2-44ae-98e1-98537991f948 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:05:03.809358 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1445/2899] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 01:05:03 2022] POST /identity/v3/projects => generated 373 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 01:05:03.813695 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18e25d4-c56d-4be7-93d6-033a75ffb813 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:05:03.823476 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18e25d4-c56d-4be7-93d6-033a75ffb813 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:05:03.880936 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b1c1865-5a7c-42f5-ba71-7fddb6568834 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:05:03.881537 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1c1865-5a7c-42f5-ba71-7fddb6568834 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:03.881810 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1c1865-5a7c-42f5-ba71-7fddb6568834 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:03.882077 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1c1865-5a7c-42f5-ba71-7fddb6568834 admin admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:03.882697 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b1c1865-5a7c-42f5-ba71-7fddb6568834 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:05:03.884627 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b1c1865-5a7c-42f5-ba71-7fddb6568834 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:05:03.906635 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1455/2900] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 01:05:03 2022] POST /identity/v3/users => generated 287 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 01:05:03.911276 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49543e01-03a2-44ae-98e1-98537991f948 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:05:03.922610 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49543e01-03a2-44ae-98e1-98537991f948 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:05:03.927318 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52028bb3-a721-4b26-8cc4-388fddb7b0fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:05:03.927905 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52028bb3-a721-4b26-8cc4-388fddb7b0fb admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:03.928179 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52028bb3-a721-4b26-8cc4-388fddb7b0fb admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:03.928581 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52028bb3-a721-4b26-8cc4-388fddb7b0fb admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:03.929227 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52028bb3-a721-4b26-8cc4-388fddb7b0fb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:05:03.930823 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52028bb3-a721-4b26-8cc4-388fddb7b0fb admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:05:03.940650 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1446/2901] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 01:05:03 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 01:05:03.944968 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b1c1865-5a7c-42f5-ba71-7fddb6568834 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:05:03.955777 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b1c1865-5a7c-42f5-ba71-7fddb6568834 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:05:03.960478 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bedd7566-cf5f-44f3-93c0-7e2442dcc879 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:05:03.961068 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bedd7566-cf5f-44f3-93c0-7e2442dcc879 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:03.961346 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bedd7566-cf5f-44f3-93c0-7e2442dcc879 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:03.961615 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bedd7566-cf5f-44f3-93c0-7e2442dcc879 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:03.962249 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bedd7566-cf5f-44f3-93c0-7e2442dcc879 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:05:03.963901 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bedd7566-cf5f-44f3-93c0-7e2442dcc879 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:05:03.975110 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1456/2902] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 01:05:03 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 01:05:03.979313 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52028bb3-a721-4b26-8cc4-388fddb7b0fb admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:05:03.989927 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52028bb3-a721-4b26-8cc4-388fddb7b0fb admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:05:03.994568 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05eacc51-7a38-430e-9bcf-49e6e607e64a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:05:03.995240 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05eacc51-7a38-430e-9bcf-49e6e607e64a admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:03.995518 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05eacc51-7a38-430e-9bcf-49e6e607e64a admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:03.995788 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05eacc51-7a38-430e-9bcf-49e6e607e64a admin admin] PATH_INFO: `/v3/projects/74dc2b85295647929cd675c10ea882c7/users/f63f7c17ba29451b8ba9599636c17e67/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:04.010375 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05eacc51-7a38-430e-9bcf-49e6e607e64a admin admin] RBAC: Authorizing `identity:create_grant(project_id=74dc2b85295647929cd675c10ea882c7, user_id=f63f7c17ba29451b8ba9599636c17e67, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:05:04.013632 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05eacc51-7a38-430e-9bcf-49e6e607e64a admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:05:04.026337 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1447/2903] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 01:05:03 2022] PUT /identity/v3/projects/74dc2b85295647929cd675c10ea882c7/users/f63f7c17ba29451b8ba9599636c17e67/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 01:05:04.031999 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23a392a-c7ce-45d9-9695-f5daf71c2238 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:04.032481 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23a392a-c7ce-45d9-9695-f5daf71c2238 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:04.032598 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23a392a-c7ce-45d9-9695-f5daf71c2238 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:04.051478 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-e23a392a-c7ce-45d9-9695-f5daf71c2238 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:05:04.052650 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-e23a392a-c7ce-45d9-9695-f5daf71c2238 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:05:04.116759 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1457/2904] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 01:05:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 01:05:04.126438 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05eacc51-7a38-430e-9bcf-49e6e607e64a admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:05:04.128155 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-05eacc51-7a38-430e-9bcf-49e6e607e64a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:05:04.142764 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:DELETE=9,keystone:SELECT=87,keystone:INSERT=4 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:05:04.186982 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05eacc51-7a38-430e-9bcf-49e6e607e64a admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:05:04.190841 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee3536a3-922a-4462-bfdc-cdf1751961e2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:05:04.191441 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3536a3-922a-4462-bfdc-cdf1751961e2 service neutron] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:04.191716 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3536a3-922a-4462-bfdc-cdf1751961e2 service neutron] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:04.191984 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3536a3-922a-4462-bfdc-cdf1751961e2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:04.201631 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee3536a3-922a-4462-bfdc-cdf1751961e2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:05:04.203088 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee3536a3-922a-4462-bfdc-cdf1751961e2 service neutron] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:05:04.212295 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1448/2905] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 01:05:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 01:05:04.823725 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832ebc81-916f-4a80-bee7-592113638ad3 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:04.824014 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832ebc81-916f-4a80-bee7-592113638ad3 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:04.824308 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832ebc81-916f-4a80-bee7-592113638ad3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:04.853462 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-832ebc81-916f-4a80-bee7-592113638ad3 None None] MFA Rules not processed for user `f63f7c17ba29451b8ba9599636c17e67`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:05:04.854603 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-832ebc81-916f-4a80-bee7-592113638ad3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:05:04.872179 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:05:04.923960 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1458/2906] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 01:05:04 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 01:05:04.935255 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee3536a3-922a-4462-bfdc-cdf1751961e2 service neutron] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:05:04.936917 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-ee3536a3-922a-4462-bfdc-cdf1751961e2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:05:05.001106 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee3536a3-922a-4462-bfdc-cdf1751961e2 service neutron] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:05:05.004880 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce17b10a-6cbc-411b-a580-3bd3fe58434b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22374eaff0834245b88506dcd7b24ede', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:05:05.005489 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce17b10a-6cbc-411b-a580-3bd3fe58434b service cinder] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:05.005768 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce17b10a-6cbc-411b-a580-3bd3fe58434b service cinder] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:05.006039 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce17b10a-6cbc-411b-a580-3bd3fe58434b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:05.015183 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce17b10a-6cbc-411b-a580-3bd3fe58434b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:05:05.016619 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce17b10a-6cbc-411b-a580-3bd3fe58434b service cinder] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:05:05.025433 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1449/2907] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 01:05:04 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 01:05:06.387591 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d3a282-f795-4c58-9bd6-79d924791b6f None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:06.388593 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d3a282-f795-4c58-9bd6-79d924791b6f None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:06.388878 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d3a282-f795-4c58-9bd6-79d924791b6f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:06.407950 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-c2d3a282-f795-4c58-9bd6-79d924791b6f None None] MFA Rules not processed for user `f63f7c17ba29451b8ba9599636c17e67`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:05:06.409397 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-c2d3a282-f795-4c58-9bd6-79d924791b6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:05:06.416163 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1459/2908] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 01:05:06 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 01:05:06.654173 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce17b10a-6cbc-411b-a580-3bd3fe58434b service cinder] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:05:06.655588 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-ce17b10a-6cbc-411b-a580-3bd3fe58434b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:05:06.667448 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce17b10a-6cbc-411b-a580-3bd3fe58434b service cinder] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:05:06.671196 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3a6139e-53bb-4f6b-b7d9-78bef70c9fbb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:05:06.671776 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a6139e-53bb-4f6b-b7d9-78bef70c9fbb service neutron] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:06.672048 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a6139e-53bb-4f6b-b7d9-78bef70c9fbb service neutron] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:06.672313 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a6139e-53bb-4f6b-b7d9-78bef70c9fbb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:06.673936 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-e3a6139e-53bb-4f6b-b7d9-78bef70c9fbb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:05:06.682966 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3a6139e-53bb-4f6b-b7d9-78bef70c9fbb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:05:06.684370 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3a6139e-53bb-4f6b-b7d9-78bef70c9fbb service neutron] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:05:06.693817 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1450/2909] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 01:05:06 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 01:05:16.401980 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=22 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:05:16.689420 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=52 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:05:20.762135 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96398efa-a36e-4eca-b744-c2d16e50acf7 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:20.762689 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96398efa-a36e-4eca-b744-c2d16e50acf7 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:20.762689 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96398efa-a36e-4eca-b744-c2d16e50acf7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:20.781372 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-96398efa-a36e-4eca-b744-c2d16e50acf7 None None] MFA Rules not processed for user `f63f7c17ba29451b8ba9599636c17e67`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:05:20.782572 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-96398efa-a36e-4eca-b744-c2d16e50acf7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:05:20.789675 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1460/2910] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 01:05:20 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 01:05:21.015803 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3a6139e-53bb-4f6b-b7d9-78bef70c9fbb service neutron] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:05:21.024512 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3a6139e-53bb-4f6b-b7d9-78bef70c9fbb service neutron] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:05:21.028054 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d11d371d-7a5e-41ee-9d10-41859a1f35a8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:05:21.029271 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d11d371d-7a5e-41ee-9d10-41859a1f35a8 service neutron] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:21.029543 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d11d371d-7a5e-41ee-9d10-41859a1f35a8 service neutron] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:21.029809 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d11d371d-7a5e-41ee-9d10-41859a1f35a8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:21.031495 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-d11d371d-7a5e-41ee-9d10-41859a1f35a8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:05:21.041124 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d11d371d-7a5e-41ee-9d10-41859a1f35a8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:05:21.042532 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d11d371d-7a5e-41ee-9d10-41859a1f35a8 service neutron] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:05:21.051081 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1451/2911] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 01:05:21 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 01:05:30.776109 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=3 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:05:31.047431 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=6 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:05:39.348306 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52d2c05-cf3b-4eec-9309-b245e80969c9 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:39.348807 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52d2c05-cf3b-4eec-9309-b245e80969c9 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:39.348930 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52d2c05-cf3b-4eec-9309-b245e80969c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:39.400966 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-e52d2c05-cf3b-4eec-9309-b245e80969c9 None None] MFA Rules not processed for user `9fc4274836844ae4ad5d16d74c36f671`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:05:39.402090 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-e52d2c05-cf3b-4eec-9309-b245e80969c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:05:39.467797 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1461/2912] 10.222.0.27 () {58 vars in 1009 bytes} [Sun Sep 11 01:05:39 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 01:05:39.513429 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d11d371d-7a5e-41ee-9d10-41859a1f35a8 service neutron] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:05:39.514841 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-d11d371d-7a5e-41ee-9d10-41859a1f35a8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:05:39.603319 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d11d371d-7a5e-41ee-9d10-41859a1f35a8 service neutron] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:05:39.609815 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0abdb6b5-2b83-4100-b00a-eb10deaab3ea service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bcb73dec73464a3abc4b774abb4f416f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:05:39.610420 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abdb6b5-2b83-4100-b00a-eb10deaab3ea service swift] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:39.610692 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abdb6b5-2b83-4100-b00a-eb10deaab3ea service swift] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:39.611008 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abdb6b5-2b83-4100-b00a-eb10deaab3ea service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:39.619259 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0abdb6b5-2b83-4100-b00a-eb10deaab3ea service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:05:39.620819 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0abdb6b5-2b83-4100-b00a-eb10deaab3ea service swift] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:05:39.629661 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1452/2913] 10.222.0.27 () {60 vars in 1384 bytes} [Sun Sep 11 01:05:39 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 11 01:05:49.469699 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=40 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:05:49.626676 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=41 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:05:55.294044 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493d3992-d607-4b57-8c7c-02fad4a8a790 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:55.294546 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493d3992-d607-4b57-8c7c-02fad4a8a790 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:55.294690 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493d3992-d607-4b57-8c7c-02fad4a8a790 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:55.314252 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-493d3992-d607-4b57-8c7c-02fad4a8a790 None None] MFA Rules not processed for user `9fc4274836844ae4ad5d16d74c36f671`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:05:55.315491 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-493d3992-d607-4b57-8c7c-02fad4a8a790 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:05:55.322481 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1462/2914] 10.222.0.27 () {58 vars in 967 bytes} [Sun Sep 11 01:05:55 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 11 01:05:55.334628 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0abdb6b5-2b83-4100-b00a-eb10deaab3ea service swift] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:05:55.345215 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0abdb6b5-2b83-4100-b00a-eb10deaab3ea service swift] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:05:55.350175 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08282e4c-c0d0-4a18-853f-03bdc13de35d service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bcb73dec73464a3abc4b774abb4f416f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:05:55.350835 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08282e4c-c0d0-4a18-853f-03bdc13de35d service swift] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:55.351117 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08282e4c-c0d0-4a18-853f-03bdc13de35d service swift] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:55.351386 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08282e4c-c0d0-4a18-853f-03bdc13de35d service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:55.360036 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08282e4c-c0d0-4a18-853f-03bdc13de35d service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:05:55.361574 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08282e4c-c0d0-4a18-853f-03bdc13de35d service swift] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:05:55.369384 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1453/2915] 10.222.0.27 () {60 vars in 1384 bytes} [Sun Sep 11 01:05:55 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 11 01:05:57.269192 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-493d3992-d607-4b57-8c7c-02fad4a8a790 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:05:57.281690 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-493d3992-d607-4b57-8c7c-02fad4a8a790 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:05:57.286693 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046d11cf-4513-45fc-adc4-9d1266edcfa0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:05:57.287402 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046d11cf-4513-45fc-adc4-9d1266edcfa0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:57.287682 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046d11cf-4513-45fc-adc4-9d1266edcfa0 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:57.287956 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046d11cf-4513-45fc-adc4-9d1266edcfa0 admin admin] PATH_INFO: `/v3/users/f63f7c17ba29451b8ba9599636c17e67` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:57.289115 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-046d11cf-4513-45fc-adc4-9d1266edcfa0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f63f7c17ba29451b8ba9599636c17e67)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:05:57.291419 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-046d11cf-4513-45fc-adc4-9d1266edcfa0 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:05:57.337258 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-046d11cf-4513-45fc-adc4-9d1266edcfa0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f63f7c17ba29451b8ba9599636c17e67'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:05:57.337919 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-046d11cf-4513-45fc-adc4-9d1266edcfa0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f63f7c17ba29451b8ba9599636c17e67'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:05:57.338529 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-046d11cf-4513-45fc-adc4-9d1266edcfa0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f63f7c17ba29451b8ba9599636c17e67'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:05:57.346179 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-046d11cf-4513-45fc-adc4-9d1266edcfa0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f63f7c17ba29451b8ba9599636c17e67'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:05:57.357960 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-046d11cf-4513-45fc-adc4-9d1266edcfa0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f63f7c17ba29451b8ba9599636c17e67'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:05:57.371305 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1463/2916] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 01:05:57 2022] DELETE /identity/v3/users/f63f7c17ba29451b8ba9599636c17e67 => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 01:05:57.543164 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08282e4c-c0d0-4a18-853f-03bdc13de35d service swift] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:05:57.554482 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08282e4c-c0d0-4a18-853f-03bdc13de35d service swift] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:05:57.617279 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8557398-3567-43a7-ac93-3ade3d9f72f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:05:57.617900 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8557398-3567-43a7-ac93-3ade3d9f72f7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:57.618195 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8557398-3567-43a7-ac93-3ade3d9f72f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:57.618447 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8557398-3567-43a7-ac93-3ade3d9f72f7 admin admin] PATH_INFO: `/v3/projects/74dc2b85295647929cd675c10ea882c7` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:57.619629 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8557398-3567-43a7-ac93-3ade3d9f72f7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=74dc2b85295647929cd675c10ea882c7)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:05:57.621607 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8557398-3567-43a7-ac93-3ade3d9f72f7 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:05:57.670899 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-a8557398-3567-43a7-ac93-3ade3d9f72f7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '74dc2b85295647929cd675c10ea882c7', 'request_id': 'req-a8557398-3567-43a7-ac93-3ade3d9f72f7'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:05:57.677857 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-a8557398-3567-43a7-ac93-3ade3d9f72f7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '74dc2b85295647929cd675c10ea882c7', 'request_id': 'req-a8557398-3567-43a7-ac93-3ade3d9f72f7'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:05:57.683774 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-a8557398-3567-43a7-ac93-3ade3d9f72f7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '74dc2b85295647929cd675c10ea882c7', 'request_id': 'req-a8557398-3567-43a7-ac93-3ade3d9f72f7'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:05:57.696586 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-a8557398-3567-43a7-ac93-3ade3d9f72f7 admin admin] The token cache is being invalidate because project 74dc2b85295647929cd675c10ea882c7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87923) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 01:05:57.696872 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-a8557398-3567-43a7-ac93-3ade3d9f72f7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:05:57.697497 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-a8557398-3567-43a7-ac93-3ade3d9f72f7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:05:57.699039 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1454/2917] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 01:05:57 2022] DELETE /identity/v3/projects/74dc2b85295647929cd675c10ea882c7 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 01:05:57.704565 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec98f14-669f-4a72-947e-f6fb10273119 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:57.704844 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec98f14-669f-4a72-947e-f6fb10273119 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:57.705117 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec98f14-669f-4a72-947e-f6fb10273119 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:57.726194 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-cec98f14-669f-4a72-947e-f6fb10273119 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:05:57.727437 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-cec98f14-669f-4a72-947e-f6fb10273119 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:05:57.788788 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1464/2918] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 01:05:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 01:05:57.797151 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62764ba-8bdd-4965-b6fc-358c9cdec570 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:57.797440 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62764ba-8bdd-4965-b6fc-358c9cdec570 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:57.797715 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62764ba-8bdd-4965-b6fc-358c9cdec570 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:57.819201 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-e62764ba-8bdd-4965-b6fc-358c9cdec570 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:05:57.820327 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-e62764ba-8bdd-4965-b6fc-358c9cdec570 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:05:57.828972 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1455/2919] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 01:05:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 01:05:57.833242 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec98f14-669f-4a72-947e-f6fb10273119 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:05:57.843020 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec98f14-669f-4a72-947e-f6fb10273119 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:05:57.847683 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21fe3ed1-ea6a-44a9-bc16-6876e599c19c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:05:57.848283 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fe3ed1-ea6a-44a9-bc16-6876e599c19c admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:57.848674 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fe3ed1-ea6a-44a9-bc16-6876e599c19c admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:57.848854 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fe3ed1-ea6a-44a9-bc16-6876e599c19c admin admin] PATH_INFO: `/v3/domains` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:57.849469 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21fe3ed1-ea6a-44a9-bc16-6876e599c19c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=87924) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 11 01:05:57.849830 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21fe3ed1-ea6a-44a9-bc16-6876e599c19c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:05:57.851414 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21fe3ed1-ea6a-44a9-bc16-6876e599c19c admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:05:57.863385 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1465/2920] 10.222.0.27 () {58 vars in 1153 bytes} [Sun Sep 11 01:05:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 11 01:05:57.867725 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e62764ba-8bdd-4965-b6fc-358c9cdec570 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:05:57.877701 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e62764ba-8bdd-4965-b6fc-358c9cdec570 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:05:57.882291 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9df7852c-3342-452a-a379-29ba4431f5c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:05:57.882906 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df7852c-3342-452a-a379-29ba4431f5c7 admin admin] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:57.883179 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df7852c-3342-452a-a379-29ba4431f5c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:57.883447 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df7852c-3342-452a-a379-29ba4431f5c7 admin admin] PATH_INFO: `/v3/projects` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:57.884073 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9df7852c-3342-452a-a379-29ba4431f5c7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:05:57.885943 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9df7852c-3342-452a-a379-29ba4431f5c7 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:05:57.916796 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1456/2921] 10.222.0.27 () {60 vars in 1153 bytes} [Sun Sep 11 01:05:57 2022] POST /identity/v3/projects => generated 369 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 01:05:57.921055 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21fe3ed1-ea6a-44a9-bc16-6876e599c19c admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:05:57.931093 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21fe3ed1-ea6a-44a9-bc16-6876e599c19c admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:05:57.979484 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:05:57.989949 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-514b2bc3-566f-4948-af5f-12528a740963 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:05:57.990550 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514b2bc3-566f-4948-af5f-12528a740963 admin admin] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:57.990852 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514b2bc3-566f-4948-af5f-12528a740963 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:57.991124 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514b2bc3-566f-4948-af5f-12528a740963 admin admin] PATH_INFO: `/v3/users` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:57.991752 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-514b2bc3-566f-4948-af5f-12528a740963 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:05:57.993659 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-514b2bc3-566f-4948-af5f-12528a740963 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:05:58.014768 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1466/2922] 10.222.0.27 () {60 vars in 1144 bytes} [Sun Sep 11 01:05:57 2022] POST /identity/v3/users => generated 285 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 11 01:05:58.019157 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9df7852c-3342-452a-a379-29ba4431f5c7 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:05:58.029656 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9df7852c-3342-452a-a379-29ba4431f5c7 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:05:58.034539 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1671bea5-810b-4380-aa88-05a650b2b1ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:05:58.035161 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1671bea5-810b-4380-aa88-05a650b2b1ea admin admin] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:58.035437 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1671bea5-810b-4380-aa88-05a650b2b1ea admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:58.035703 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1671bea5-810b-4380-aa88-05a650b2b1ea admin admin] PATH_INFO: `/v3/roles` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:58.036337 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1671bea5-810b-4380-aa88-05a650b2b1ea admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:05:58.037922 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1671bea5-810b-4380-aa88-05a650b2b1ea admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:05:58.048415 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1457/2923] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 01:05:58 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 01:05:58.052449 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-514b2bc3-566f-4948-af5f-12528a740963 admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:05:58.063530 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-514b2bc3-566f-4948-af5f-12528a740963 admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:05:58.068334 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec6025c-766c-449d-973b-7aa654594e68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:05:58.068945 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec6025c-766c-449d-973b-7aa654594e68 admin admin] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:58.069376 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec6025c-766c-449d-973b-7aa654594e68 admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:58.069618 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec6025c-766c-449d-973b-7aa654594e68 admin admin] PATH_INFO: `/v3/roles` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:58.070255 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ec6025c-766c-449d-973b-7aa654594e68 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:05:58.071856 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ec6025c-766c-449d-973b-7aa654594e68 admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:05:58.082481 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1467/2924] 10.222.0.27 () {58 vars in 1122 bytes} [Sun Sep 11 01:05:58 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 11 01:05:58.086499 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1671bea5-810b-4380-aa88-05a650b2b1ea admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:05:58.096698 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1671bea5-810b-4380-aa88-05a650b2b1ea admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:05:58.101717 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d0a43d-5624-4c64-bb24-cb38f4fc0f53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:05:58.102596 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d0a43d-5624-4c64-bb24-cb38f4fc0f53 admin admin] REQUEST_METHOD: `PUT` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:58.102988 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d0a43d-5624-4c64-bb24-cb38f4fc0f53 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:58.103271 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d0a43d-5624-4c64-bb24-cb38f4fc0f53 admin admin] PATH_INFO: `/v3/projects/a283581147f44570a4ab86a72ecc7392/users/95a57659a315480f8d149677f51156e9/roles/2e64fc597b6447df99a7992d411d1a34` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:58.119510 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57d0a43d-5624-4c64-bb24-cb38f4fc0f53 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a283581147f44570a4ab86a72ecc7392, user_id=95a57659a315480f8d149677f51156e9, role_id=2e64fc597b6447df99a7992d411d1a34)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:05:58.122155 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57d0a43d-5624-4c64-bb24-cb38f4fc0f53 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:05:58.135969 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1458/2925] 10.222.0.27 () {60 vars in 1483 bytes} [Sun Sep 11 01:05:58 2022] PUT /identity/v3/projects/a283581147f44570a4ab86a72ecc7392/users/95a57659a315480f8d149677f51156e9/roles/2e64fc597b6447df99a7992d411d1a34 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 11 01:05:58.147577 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d636e5a8-c99d-47c5-add8-65e37c167144 None None] REQUEST_METHOD: `POST` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:58.147908 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d636e5a8-c99d-47c5-add8-65e37c167144 None None] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:58.148163 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d636e5a8-c99d-47c5-add8-65e37c167144 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:58.177181 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.auth.core [None req-d636e5a8-c99d-47c5-add8-65e37c167144 None None] MFA Rules not processed for user `95a57659a315480f8d149677f51156e9`. Rule list: `[]` (Enabled: `True`). {{(pid=87924) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:05:58.178379 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-d636e5a8-c99d-47c5-add8-65e37c167144 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:05:58.240643 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1468/2926] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 01:05:58 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 01:05:58.253123 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d0a43d-5624-4c64-bb24-cb38f4fc0f53 admin admin] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:05:58.254763 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-57d0a43d-5624-4c64-bb24-cb38f4fc0f53 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:05:58.272133 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:05:58.334430 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d0a43d-5624-4c64-bb24-cb38f4fc0f53 admin admin] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:05:58.338094 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d88ed9-4f71-4539-9ba4-88c1045f1796 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '06b1c911731f4267a5f37e91d948e7e9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:05:58.338697 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d88ed9-4f71-4539-9ba4-88c1045f1796 service glance] REQUEST_METHOD: `GET` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:05:58.339007 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d88ed9-4f71-4539-9ba4-88c1045f1796 service glance] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:05:58.339284 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d88ed9-4f71-4539-9ba4-88c1045f1796 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:05:58.348334 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8d88ed9-4f71-4539-9ba4-88c1045f1796 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:05:58.349728 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8d88ed9-4f71-4539-9ba4-88c1045f1796 service glance] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:05:58.359148 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1459/2927] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 01:05:58 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 01:06:08.237770 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=42,keystone:INSERT=3 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:06:08.354541 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=29 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:06:28.571479 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d636e5a8-c99d-47c5-add8-65e37c167144 None None] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:06:28.635287 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d636e5a8-c99d-47c5-add8-65e37c167144 None None] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:06:28.639801 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc1108b-b293-4fc7-9b31-4ae8c378d4fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:06:28.640425 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc1108b-b293-4fc7-9b31-4ae8c378d4fa admin admin] REQUEST_METHOD: `DELETE` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:06:28.640704 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc1108b-b293-4fc7-9b31-4ae8c378d4fa admin admin] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:06:28.640979 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc1108b-b293-4fc7-9b31-4ae8c378d4fa admin admin] PATH_INFO: `/v3/users/95a57659a315480f8d149677f51156e9` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:06:28.642086 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fc1108b-b293-4fc7-9b31-4ae8c378d4fa admin admin] RBAC: Authorizing `identity:delete_user(user_id=95a57659a315480f8d149677f51156e9)` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:06:28.644035 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fc1108b-b293-4fc7-9b31-4ae8c378d4fa admin admin] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:06:28.689947 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-7fc1108b-b293-4fc7-9b31-4ae8c378d4fa admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '95a57659a315480f8d149677f51156e9'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:06:28.690552 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-7fc1108b-b293-4fc7-9b31-4ae8c378d4fa admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '95a57659a315480f8d149677f51156e9'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:06:28.691184 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-7fc1108b-b293-4fc7-9b31-4ae8c378d4fa admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '95a57659a315480f8d149677f51156e9'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:06:28.697972 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-7fc1108b-b293-4fc7-9b31-4ae8c378d4fa admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '95a57659a315480f8d149677f51156e9'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:06:28.709580 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.notifications [None req-7fc1108b-b293-4fc7-9b31-4ae8c378d4fa admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '95a57659a315480f8d149677f51156e9'} {{(pid=87924) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:06:28.722304 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1469/2928] 10.222.0.27 () {58 vars in 1224 bytes} [Sun Sep 11 01:06:28 2022] DELETE /identity/v3/users/95a57659a315480f8d149677f51156e9 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 01:06:28.726861 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-437ee31e-2691-4695-8d6c-40128eb53610 None None] REQUEST_METHOD: `POST` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:06:28.727379 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-437ee31e-2691-4695-8d6c-40128eb53610 None None] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:06:28.727495 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-437ee31e-2691-4695-8d6c-40128eb53610 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:06:28.746930 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.auth.core [None req-437ee31e-2691-4695-8d6c-40128eb53610 None None] MFA Rules not processed for user `36edaa883d654921ac17e7745c0ff320`. Rule list: `[]` (Enabled: `True`). {{(pid=87923) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 11 01:06:28.748122 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-437ee31e-2691-4695-8d6c-40128eb53610 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:06:28.805871 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1460/2929] 10.222.0.27 () {58 vars in 958 bytes} [Sun Sep 11 01:06:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 11 01:06:28.816215 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc1108b-b293-4fc7-9b31-4ae8c378d4fa admin admin] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:06:28.817613 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-7fc1108b-b293-4fc7-9b31-4ae8c378d4fa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:06:28.873433 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc1108b-b293-4fc7-9b31-4ae8c378d4fa admin admin] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:06:28.877703 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d1c887-e11b-4330-90bc-7ade85ef333b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a5f7b13283cb4ebfb25160aa6ad0e2f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:06:28.878297 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d1c887-e11b-4330-90bc-7ade85ef333b service neutron] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:06:28.878573 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d1c887-e11b-4330-90bc-7ade85ef333b service neutron] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:06:28.878874 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d1c887-e11b-4330-90bc-7ade85ef333b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:06:28.887871 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18d1c887-e11b-4330-90bc-7ade85ef333b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:06:28.889276 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18d1c887-e11b-4330-90bc-7ade85ef333b service neutron] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:06:28.898598 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1470/2930] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 01:06:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 01:06:29.165100 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-437ee31e-2691-4695-8d6c-40128eb53610 None None] Authenticating user token {{(pid=87923) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:06:29.166528 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.fernet_utils [None req-437ee31e-2691-4695-8d6c-40128eb53610 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87923) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:06:29.179481 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-437ee31e-2691-4695-8d6c-40128eb53610 None None] Validating token access rules against request {{(pid=87923) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:06:29.184548 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1068e86-28c3-4491-994a-2a54c419b133 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36edaa883d654921ac17e7745c0ff320', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '12356f1769944cc8be5c533d062adf65', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87923) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:06:29.185183 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1068e86-28c3-4491-994a-2a54c419b133 admin admin] REQUEST_METHOD: `DELETE` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:06:29.185461 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1068e86-28c3-4491-994a-2a54c419b133 admin admin] SCRIPT_NAME: `/identity` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:06:29.185731 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1068e86-28c3-4491-994a-2a54c419b133 admin admin] PATH_INFO: `/v3/projects/a283581147f44570a4ab86a72ecc7392` {{(pid=87923) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:06:29.186959 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1068e86-28c3-4491-994a-2a54c419b133 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a283581147f44570a4ab86a72ecc7392)` {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:06:29.188874 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1068e86-28c3-4491-994a-2a54c419b133 admin admin] RBAC: Authorization granted {{(pid=87923) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:06:29.238073 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-a1068e86-28c3-4491-994a-2a54c419b133 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a283581147f44570a4ab86a72ecc7392', 'request_id': 'req-a1068e86-28c3-4491-994a-2a54c419b133'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:06:29.244314 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-a1068e86-28c3-4491-994a-2a54c419b133 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a283581147f44570a4ab86a72ecc7392', 'request_id': 'req-a1068e86-28c3-4491-994a-2a54c419b133'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:06:29.249847 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-a1068e86-28c3-4491-994a-2a54c419b133 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a283581147f44570a4ab86a72ecc7392', 'request_id': 'req-a1068e86-28c3-4491-994a-2a54c419b133'} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:06:29.261915 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-a1068e86-28c3-4491-994a-2a54c419b133 admin admin] The token cache is being invalidate because project a283581147f44570a4ab86a72ecc7392 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=87923) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 11 01:06:29.262202 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-a1068e86-28c3-4491-994a-2a54c419b133 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:06:29.262840 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG keystone.notifications [None req-a1068e86-28c3-4491-994a-2a54c419b133 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=87923) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 11 01:06:29.264298 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: [pid: 87923|app: 0|req: 1461/2931] 10.222.0.27 () {58 vars in 1233 bytes} [Sun Sep 11 01:06:29 2022] DELETE /identity/v3/projects/a283581147f44570a4ab86a72ecc7392 => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 11 01:06:38.894419 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=79,keystone:DELETE=10,keystone:INSERT=1 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:06:39.258928 dv-cinder-master-3-iscsi devstack@keystone.service[87923]: DEBUG dbcounter [-] [87923] Writing DB stats keystone:SELECT=54,keystone:DELETE=7,keystone:INSERT=1 {{(pid=87923) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 11 01:06:53.818712 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d1c887-e11b-4330-90bc-7ade85ef333b service neutron] Authenticating user token {{(pid=87924) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 11 01:06:53.820140 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-18d1c887-e11b-4330-90bc-7ade85ef333b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:06:53.888923 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d1c887-e11b-4330-90bc-7ade85ef333b service neutron] Validating token access rules against request {{(pid=87924) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 11 01:06:53.948167 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5391a393-e2bb-458e-bbc3-a5ac23655407 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cdbfc2ccd73e4a5eb8c999be5b094619', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b504a5058bc6474c8cc077dfeec16799', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=87924) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 11 01:06:53.948769 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5391a393-e2bb-458e-bbc3-a5ac23655407 service placement] REQUEST_METHOD: `GET` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 11 01:06:53.949046 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5391a393-e2bb-458e-bbc3-a5ac23655407 service placement] SCRIPT_NAME: `/identity` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 11 01:06:53.949313 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5391a393-e2bb-458e-bbc3-a5ac23655407 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=87924) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 11 01:06:53.951132 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.fernet_utils [None req-5391a393-e2bb-458e-bbc3-a5ac23655407 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=87924) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 11 01:06:53.964940 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5391a393-e2bb-458e-bbc3-a5ac23655407 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 11 01:06:53.966567 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5391a393-e2bb-458e-bbc3-a5ac23655407 service placement] RBAC: Authorization granted {{(pid=87924) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 11 01:06:53.978188 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: [pid: 87924|app: 0|req: 1471/2932] 10.222.0.27 () {60 vars in 1365 bytes} [Sun Sep 11 01:06:53 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 160 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 11 01:07:03.973351 dv-cinder-master-3-iscsi devstack@keystone.service[87924]: DEBUG dbcounter [-] [87924] Writing DB stats keystone:SELECT=58 {{(pid=87924) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}